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TWO  EXTENSIONS  OF  THE  LOGARITHMIC  LEAST-SQUARES  METHOD 
FOR  ANALYZING  SUBJECTIVE  COMPARISONS 


INTRODUCTION 


1.  This  technical  memorandum  presents  two  extensions  of  the  logarithmic 
least-squares  (LLS)  method  for  analyzing  sets  of  subjective  comparisons.  The 
first  is  a  test  of  consistency  based  on  analysis-of-variance  techniques. 

Unlike  Saaty's  eigenvalue-based  consistency  ratio,  the  consistency  metric 
developed  here  does  not  depend  on  a  scale  for  eliciting  comparisons.  This 
metric  is  easily  applied  to  comparisons  sets  that  do  not  contain  comparisons 
between  all  possible  pairs  of  entities  or  that  contain  multiple  comparisons 
between  some  pairs.  Furthermore,  this  consistency  metric  provides  valuable 
insights  into  the  trade-off  between  the  number  of  comparisons  in  a  set  and  our 
ability  to  detect  inconsistency. 

2.  The  second  extension  is  a  robust  regression  approach  to  finding  a  ratio 
scale  that  best  represents  most  of  the  comparisons.  Because  the  LLS  method  is 
a  least-squares  technique,  it  is  sensitive  to  outliers,  i.e.,  comparisons  that 
contain  relatively  large  errors.  One  or  more  outliers  in  a  comparisons  set 
can  seriously  degrade  the  validity  of  the  ratio  scale  derived  from  it.  The 
robust  regression  method  selectively  de-emphasizes  outliers,  providing  a  ratio 
scale  that  best  represents  most  of  the  comparisons. 

CONTENTS 


3.  This  memorandum  defines  terminology,  then  describes  and  compares  the 
logarithmic  least-squares  and  eigenvalue  methods  for  estimating  ratio  scales 
and  assessing  consistency.  It  discusses  the  two  extensions  of  the  LLS  method 
described  above  and  presents  examples  of  their  application.  The  memorandum 
concludes  by  summarizing  the  extensions. 

DEFINITIONS 


4.  Paralleling  Crawford's  definitions  [1],  we  define  a  collection  of  n 
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entities  {E.}  and  the  existence  of  an  unknown  ratio  scale  x  =  {x..,x«,...,x  }, 

^  th  ^  ^  ^ 

where  is  the  weight  of  the  1  entity  and  x^/x^  is  the  relative  value  of 

to  E..  Ve  call  the  scale  x  the  underlying  ratio  scale  for  the  relative  values 

of  the  entities  and  refer  to  its  entries  as  weights.  Ve  also  assume  a  set  of 

comparisons 


A  =  {a.jk} 


i.J 


1, . . .  ,n 


k  =  1, . . . ,n. j 


where  a...  is  the  kt^1  comparison  between  entities  i  and  j.  A  comparison  is  an 
1 J  x 

estimate  of  the  relative  value  of  E.  and  E.,  i.e.,  of  x./x..  To  assure 

i  3  i  3 

symmetry,  we  require  that  aj^  =  l/a^ ^ •  Ve  also  require  that  a^k  he 
positive  for  all  i,  j,  and  k.  Ve  use  the  set  of  comparisons  to  estimate  the 
underlying  ratio  scale.  It  is  common  practice  to  compare  a  single  set  of 
entities  on  the  basis  of  several  attributes  to  obtain  a  ratio  scale  for  each 
attribute.  The  relative  importance  of  the  attributes  to  the  overall  goal  is 
then  determined,  and  the  ratio  scale  for  each  of  the  attributes  is  combined 
with  the  ratio  scale  for  the  overall  goal  to  provide  an  estimate  of  the  values 
of  the  entities  relative  to  the  overall  goal.  This  is  the  basis  for  Saaty's 
analytic  hierarchy  process  [2].  However,  this  memorandum  deals  only  with 
estimating  ratio  scales  from  sets  of  comparisons. 


5.  Two  methods  that  are  frequently  used  to  estimate  x  are  Saaty's  eigenvalue 
method  [2]  and  Crawford  and  Villiams'  logarithmic  least-squares  method  [3]. 
These  methods  are  discussed  below.  Both  methods  are  approaches  to  filtering 
data  in  the  comparisons  set  to  eliminate  errors  in  estimating  the  ratios. 


6.  In  addition  to  estimating  the  values  of  the  weights,  we  are  concerned  with 

the  consistency  of  the  comparisons  set.  If  A  is  a  set  of  comparisons,  A  is 

perfectly  consistent  if  a^jk  =  aiiicaljk  ^or  3»  and  1  and  all 

comparisons  of  entities  i  and  j  are  the  same,  i.e.,  a...  is  independent  of  k 

X3K 

for  all  i  and  j.  Ve  consider  randomness  as  the  antithesis  of  consistency,  and 
accordingly  test  for  consistency  by  hypothesizing  that  the  comparisons  set  is 
random  in  the  hope  of  rejecting  this  hypothesis.  If  we  cannot  reject  this 
hypothesis,  the  weights  derived  from  the  comparisons  set  are  probably  not 
meaningful  and  should  not  be  used. 
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7.  If  <  1  for  some  i  and  j,  there  is  at  least  one  pair  of  entities  for 
which  there  are  no  comparisons  in  the  set,  and  we  say  that  the  comparisons  set 
is  inco  ete.  Conversely,  if  n„  >  1  for  all  i  and  j,  the  comparisons  set  is 
complete.  Similarly,  if  n. .  >  1  for  some  i  and  j,  there  is  a  pair  of  entities 
for  which  the  set  contains  multiple  comparisons.  Thus  a  comparisons  set  may 
simultaneously  be  either  complete  or  incomplete  and  contain  multiple  compar¬ 
isons.  We  also  define  a  minimally  complete  comparisons  set  as  one  for  which 
n^  =  1  for  all  i  and  j.  For  a  minimally  complete  comparisons  set,  we  drop 
the  subscript  k  and  arrange  the  set  as  a  matrix,  referring  to  the  matrix  as  a 
(complete)  subjective  judgment  matrix  (SJM).  Because  such  a  matrix  has  only 
positive  entries  and  a.^  =  1/a^ . ,  an  SJM  is  a  positive  reciprocal  matrix.  It 
will  also  be  useful  to  refer  to  an  incomplete  comparisons  set  that  contains  no 
multiple  comparisons  as  an  incomplete  SJM.  An  SJM  must  contain  at  least  n-1 
comparisons  to  allow  us  to  compute  a  ratio  scale.  Figure  1  shows  the  rela¬ 
tionships  between  these  differing  classifications  of  comparisons  sets. 


8.  Because  a  complete  SJM  is  a  positive  reciprocal  matrix,  it  contains  N  = 
n(n-l)/2  independent  comparisons.  The  remaining  comparisons  are  dependent, 
assuring  that  the  matrix  is  positive  reciprocal.  My  convention  in  this 
memorandum  is  that  the  entries  above  the  diagonal  of  an  SJM  are  the  indep¬ 
endent  comparisons.  In  addition,  we  define  the  fraction  complete  of  an 
incomplete  SJM  as 


n 

E  _  . 

i-1  j>i  1J 


E  6.  .  /  N 


where  S.  .  equals  1  if  n. .  >1  and  0  otherwise. 
iJ  ij  " 


EIGENVALUE  (EV)  METHOD 


9.  For  complete  SJMs,  Saaty  12)  argues  that  the  best  estimate  of  the  ratio 
scale  x  is  provided  by  the  principal  right  eigenvector  of  the  SJM.  This 
vector  is  obtained  by  solving  the  matrix  equation 


Au 


A  u 
max 


(1) 
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Figure  1.  Summary  of  Terminology  for  Comparisons  Sets 


where  A  is  an  SJM,  u  is  the  principal  right  eigenvector  corresponding  to  the 
maximum  eigenvalue,  and  ^max  is  the  maximum  eigenvalue. 

10.  Saaty  also  shows  that  the  maximum  eigenvalue  is  a  measure  of  the 

consistency  of  A.  If  A  is  perfectly  consistent,  X  equals  n,  the  order  of 

max 

the  SJM.  As  A  becomes  less  consistent,  the  value  of  X  increases.  Saaty 

max  J 

uses  this  fact  to  '"instruct  a  consistency  index  for  A,  which  is  defined  as 

CI(A)  =  (X  -  n)  /  (n  -  1). 

max  v  ' 

11.  By  itself,  the  consistency  index  does  not  provide  a  basis  for  testing 
whether  A  is  random.  Saaty  proposes  using  a  consistency  ratio  as  the  basis 
for  such  a  test.  The  consistency  ratio  is 

CR( A)  =  Cl (A)  /  RI(n)  (2) 

where  CR(A)  is  the  'Consistency  ratio  for  A  and  RI(n)  is  a  random  index  for  nxn 
SJMs,  i.e.,  the  average  consistency  index  for  a  sample  of  randomly-generated 
nxn  SJMs.  The  consistency  ratio  increases  with  the  inconsistency  of  the 
comparisons  set.  Saaty  recommends  we  conclude  that  A  is  adequately  consistent 
if  its  consistency  ratio  is  less  than  0.10. 

12.  The  consistency  ratio  test  of  randomness  is  scale-dependent.  Random 
indexes  have  been  generated  for  a  variety  of  scales,  which  represent  the 
distribution  of  values  in  the  randomly-generated  matrices.  To  use  the  test 
statistic  given  in  equation  2,  a  random  index  must  be  available  for  the  same 
scale  used  to  generate  A.  The  most  popular  set  of  random  indexes  are  those 
for  Saaty's  nine-value  scale,  which  limits  the  comparisons  in  A  to  values 
between  1/9  and  9.  The  appropriateness  of  this  scale  and  whether  a  scale 
should  be  used  in  eliciting  comparisons  has  been  a  subject  of  debate  (1]. 

LOGARITHMIC  LEAST-SQUARES  METHOD 

13.  As  an  alternative  to  the  eigenvalue  method,  Crawford  and  Williams  [3j 
advocate  using  the  LLS  method  to  estimate  the  underlying  ratio  scale.  This 
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method  assumes  that  the  comparisons  are  subject  to  a  random  multiplicative 
error  e^,  i.e. , 

a. =  (x. /x. )e. ..  . 

13k  K  1  2  13k 

In  keeping  with  the  requirement  that  the  comparisons  be  reciprocally  symmet¬ 
ric,  we  require  that  the  error  distribution  also  be  reciprocally  symmetric, 
i.e.,  =  f  ( 1/ ,  where  f(x)  is  the  probability  density  function  of 

the  errors.  The  LLS  method  minimizes  the  sum  of  the  squares  of  the  logarithms 
of  the  errors,  i.e.,  it  minimizes 


SSE  =  S(e.jk^) 


where 


S(zijk)  '  Z  Z  1  zijk’ 

i_l  j>i  k=1  J 


and  Wj 


■  estimated  logarithmic  error 

=  ln  aijk  ~  ln  wi  +  ln  vy 

=  LLS  estimate  of  x.. 

1 


The  estimated  logarithmic  error  is  also  frequently  written  as 


e..,  =  y .  ..  -  b.  +  b. 
ljk  'ijk  1  j 


.here  y,j|t  -  ln  aiJk 
and  b^  *  ln  w^ . 


Equation  4  can  also  be  written  as 


*ijk-vi.  ^ijk 
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where  y. 

•"ljk 


l  b,X 
1=1 


ijkl 


i  th 


and  is  the  1  independent  variable  used  to  estimate  y^^- 

^ijkl  e<lua^s  1  if  1  =  i  *  j ,  -1  if  1  -  j  /  i,  and  zero  otherwise, 
is  the  linear  model  used  in  the  LLS  method. 


For  any  k, 
Equation  5 


14.  For  a  complete  SJM,  calculating  the  weight  of  the  i**1  entity  by  the  LLS 
method  is  equivalent  to  taking  the  geometric  mean  of  the  comparisons  in  the 
row  of  the  matrix.  (For  this  reason,  the  LLS  method  is  frequently 
referred  to  as  the  geometric  mean  (GM)  method.)  Mathematically, 


w.  = 

l 


C  ( 


n 

n 


j=l 


a.  . 
1 3 


.  1/n 


(6) 


where  c  is  an  arbitrary  constant. 

15.  In  1987,  Crawford  [1]  developed  a  test  of  consistency  for  the  LLS  method 
that  is  based  on  a  transformation  of  the  random  indexes  used  to  calculate 
Saaty's  consistency  ratio.  Hence,  this  test  is  also  scale-dependent.  Until 
Crawford  developed  his  test,  Saaty's  was  the  only  one  available.  In  some 
early  applications  of  the  LLS  method,  notably  the  State-of-the-Art  Contingency 
Analysis  (SOTACA)  model,  this  has  led  ~o  using  an  eigenvalue-based  test  of 
consistency  to  assess  the  validity  of  a  ratio  scale  generated  using  the  LLS 
method.  However,  the  EV  and  LLS  weights  are  not  the  same  except  when  the  SJM 
is  perfectly  consistent.  One  of  the  extensions  of  the  LLS  method  presented  in 
this  memorandum  is  an  analysis-of-variance  test  of  randomness  that  complements 
the  LLS  approach. 

COMPARISON  OF  EIGENVALUE  AND  LOGARITHMIC  LEAST-SQUARES  METHODS 


16.  Table  1  presents  the  advantages  and  limitations  of  the  EV  and  LLS 
methods.  The  EV  method  uses  all  the  information  in  an  SJM  to  obtain  weights, 


TABLE  1.  COMPARISON  OF  EIGENVALUE  AND 
LOGARITHMIC  LEAST-SQUARES  METHODS 


EV  Method 

LLS  Method 

Advantages 

Ratio  scale  depends  on  all 

comparisons 

Works  with  comparisons  sets  that  are 
incomplete  or  contain  multiple 

comparisons 

Limitations 

Does  not  work  with  comparisons  sets 

that  are  incomplete  or  contain 

multiple  comparisons 

Consistency  metric  is  scale-dependent 

Consistency  ratio  test  is  not  uniform 

for  SJMs  of  different  orders 

Consistency  metric  is  scale-dependent 

Ratio  scale  depends  only  on  direct 

comparisons 

Sensitive  to  outliers 
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i.e..  it  uses  the  value  of  ajjc/aj)c  to  provide  indirect  evidence  on  the  rel¬ 
ative  value  of  entities  i  and  j  in  addition  to  the  direct  evidence  provided  by 
a^j .  In  contrast,  the  LLS  method  uses  only  the  direct  comparisons  between  an 
entity  i  and  other  entities  to  determine  its  weight.  This  characteristic  is 
inherent  in  the  regression  methodology,  which  minimizes  equation  3  by  setting 
the  partial  derivative  of  SSE  with  respect  to  each  of  the  weights  equal  to 
zero.  This  approach  eliminates  consideration  of  any  comparisons  that  do  not 
involve  the  entity  whose  weight  is  being  calculated. 


17.  As  equation  1  shows,  the  EV  method  is  inherently  a  matrix  method.  This 
raises  the  question  of  how  to  proceed  if  comparisons  have  not  been  made 
between  all  pairs  of  entities.  In  contrast,  the  LLS  method  is  not  a  matrix 
method,  although  it  is  convenient  to  use  matrix-like  notation  to  remind 
ourselves  of  which  entities  are  being  compared.  As  Crawford  and  Williams  [3] 
point  out,  the  LLS  method  is  adaptable  to  comparisons  sets  that  are  incomplete 
or  contain  several  comparisons  between  a  pair  of  entities. 

18.  An  additional  limitation  of  the  EV  method  is  that  it  does  not  provide  a 
uniform  criterion  for  testing  the  hypothesis  of  randomness  as  the  order  of  the 
matrix  changes.  Saaty's  consistency  test  is  based  on  the  value  of  the  consis¬ 
tency  ratio  itself  rather  than  its  probability  distribution.  In  [2],  Saaty 
states  that  for  random  matrices  of  orders  4  through  9,  the  maximum  eigen¬ 
values,  and  hence  the  random  indexes,  approximately  follow  a  truncated  normal 
distribution.  He  also  gives  values  for  the  means  and  variances  of  the  maximum 
eigenvalues.  These  are  shown  in  Table  2,  which  also  gives  the  probability, 
based  on  the  normal  distribution,  that  the  consistency  ratio  of  a  random 
matrix  is  less  than  0.10.  This  probability  is  given  by 


«xs  -  (P(u)  -  P(n))  /  (1  -  P(n) ) 
where  u  *  n  +  0.1  CI(n)  (n  -  1) 


and  P(x)  is  the  cumulative  normal  probability  distibution  with  the  mean  and 
variance  taken  from  Table  2.  We  will  refer  to  a<,  as  Saaty's  risk.  Saaty's 
risk  is  the  probability  of  falsely  rejecting  the  hypothesis  that  the  matrix  is 
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random,  i.e.,  the  probability  we  will  conclude  that  A  is  non-random  when  in 
fact  it  is.  Table  2  shows  that  as  the  order  of  the  matrix  increases,  the 
consistency  ratio  test  imposes  increasingly  stringent  criteria  for  rejecting  a 
random  matrix.  This  may  be  undesirable  because  the  higher  number  of  compar¬ 
isons  in  larger  SJMs  make  consistency  harder  to  attain,  suggesting  that  a  more 
lenient  test  rather  than  a  more  stringent  one  would  be  appropriate  as  SJM 
order  increases. 

Table  2.  Effect  of  Matrix  Order  on 
Probability  of  Falsely  Rejecting  a  Random  Matrix 


Matrix 

Maximum  Eigenvalue 

Probability  of 

Order 

Mean 

Variance 

False  Rejectio 

4 

6.650 

3.370 

0.025 

5 

9.418 

4.424 

0.011 

6 

12.313 

4.413 

0.002 

7 

15.000 

4.123 

<  0.001 

8 

17.952 

3.627 

«  0.001 

9 

20.565 

3.327 

«  0.001 

EXTENSIONS  OF  THE  LOGARITHMIC  LEAST-SQUARES  METHOD 

19.  This  memorandum  presents  two  extensions  of  the  LLS  method:  a  scale- 
independent  test  for  the  randomness  of  comparisons,  and  a  robust  regression 
method  for  finding  weights  that  best  represent  most  of  comparisons. 

Scale-Independent  Test  of  Randomness 

20.  As  shown  above,  the  LLS  method  for  determining  weights  is  based  on 
log-linear  regression  of  the  comparisons  set.  This  raises  the  possibility  of 
using  analysis-of-variance  techniques  to  formulate  a  test  statistic  for  the 
randomness  of  a  comparisons  set.  As  Appendix  A  shows,  formulating  such  a  test 
statistic  is  possible.  However,  it  is  not  the  test  statistic  used  for  more 
general  regression  models.  The  general  linear  regression  model  is 
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P-1 

*i  +  ei 

k=0 


where  y.  =  the  variable  being  fitted, 
i  th 

^  *  the  ll  regression  coefficient, 

=  the  lt^  independent  variable  for  dependent  variable  y^, 

X^q  =  1  for  all  i, 

and  e.  =  error  term. 

1 

For  this  model,  the  total  sum  of  squares  (SSTO),  defined  as 


SSTO  =  £  (y.  -  y)2, 
i=l 


can  be  partitioned  into  error  and  regression  sums  of  squares.  These  are 


n 

SSE  =  £  (y.  -  y.) 
i=l 


2 


and 


SSR  =  Z  <y.  -  y ), 
i=l 

respectively.  Their  mean  squares  are 
MSE  =  SSE  /  (n-p) 

and 

HSR  =  SSR  /  (p-1), 
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where  p  is  the  number  of  regression  coefficients  estimated.  From  these 
values,  we  define  a  test  statistic  F*: 

F*  =  MSR  /  MSE. 


2 

The  expected  value  of  MSE  is  a  ,  the  variance  of  the  error  terms.  The 

2 

expected  value  of  MSR  is  a  plus  a  function  of  the  estimated  weights.  If  the 
dependent  variables  y\  are  random  variations  around  a  constant  mean, 

are  all  zero,  and  MSR  equals  MSE.  Hence  we  expect  that  if  the 
y^'s  are  random,  F*  will  be  near  1.  In  addition,  if  the  errors  are 
normally  distributed,  F*  will  be  F-distributed  with  (p-1,  n-p)  degrees  of 
freedom.  In  this  case,  the  probability  that  a  set  of  dependent  variables  is 
random  is 

F* 

s  =  J  f (x;  p-1,  n-p)  dx  (8) 

0 


where  f(x;  p-1,  n-p)  is  the  probability  density  function  of  the  F  distribu¬ 
tion.  The  probability  a  of  falsely  rejecting  the  hypothesis  of  randomness  is 

a  «  1  -  s.  (9) 

21.  Appendix  A  shows  that  the  general  regression  model  of  equation  7  is  in¬ 
appropriate  for  estimating  weights  from  sets  of  comparisons.  Unlike  the 
general  regression  model,  the  LLS  model  of  equation  5  has  no  constant  term. 
This  term's  presence  assures  that  the  total  sum  of  squares  can  be  separated 
into  error  and  regression  sums  of  squares.  Its  absence  in  equation  5  pre¬ 
cludes  using  the  F*  statistic  to  test  whether  an  SJM  is  random.  However, 
Appendix  A  shows  that,  for  the  LLS  model,  we  can  formulate  an  alternative  test 
statistic  that  is  similar  to  F*.  This  statistic,  denoted  F*y,  is  given  by 

F*u  =  MSRU  /  MSE 

where  MSRU  is  the  uncorrected  regression  mean  square  given  by 
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MSRU  =  SSRU  /  n 


and 


SSRU  =  S(y..2).  (10) 

If  the  error  terms  are  normally  distributed,  F*^  will  be  F-distributed  with 
(n,  N-n)  degrees  of  freedom.  The  significance  level  and  risk,  for  the 
hypothesis  of  randomness  can  then  be  determined  as  in  equations  8  and  9. 

22.  Both  SSE  and  SSRU  have  intuitively  appealing  interpretations.  SSE  is  the 
distance,  in  logari thms-of-errors  terms,  between  the  SJM  being  analyzed  and 
the  nearest  consistent  matrix,  whose  entries  are  w^/v^.  Similarly,  SSRU  is 
the  distance,  in  the  same  terms,  between  the  nearest  consistent  matrix  and  a 
unit  matrix,  i.e.,  one  having  all  its  entries  equal  to  1.  If  the  comparisons 
in  the  SJM  are  random,  we  expect  the  nearest  consistent  matrix  to  be  nearly  a 
unit  matrix,  indicating  that  the  weights  of  all  the  entities  are  equal. 
Although  there  is  nothing  inherently  wrong  with  equal  weights,  generating 
comparisons  that  provide  them  is  easy  and  accurate.  In  contrast,  we  expect 
a  randomly-generated  SJM  to  have  a  high  value  of  MSE  and  hence  a  low  value  of 

r  U 


23.  The  consistency  test  based  on  F*y  is  distribution-dependent,  i.e.,  it 
assumes  that  the  distribution  of  the  errors  is  log-normal  or  equivalently 
that  the  distribution  of  the  logarithms  of  the  errors  e...  is  normal.  This 
assumption  is  easy  to  test.  Neter  and  Vasserman  [4]  state  that  the  F  test  is 
fairly  robust  in  tolerating  departures  of  the  errors  from  normality,  and  that 
it  is  most  sensitive  to  the  skewness  of  the  error  distribution.  Accordingly, 
it  is  best  to  test  the  assumption  that  errors  are  normally  distributed  using 
the  skewness  of  the  observed  error  distribution.  Appendix  B  presents  stat¬ 
istics  for  testing  the  normality  of  the  error  distribution  based  on  its 
skewness.  These  statistics  are  based  on  Monte  Carlo  simulation  results  and 
extend  the  skewness  statistics  in  the  Biometrika  tables  [5]  to  sample  sizes  as 
small  as  3.  The  statistics  in  Appendix  B  are  for  SJMs  of  orders  3  through  22. 
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Statistics  for  incomplete  comparison  sets  can  be  interpolated  from  the  tables 
in  Appendix  B. 

24.  Even  if  the  assumption  of  normally-distributed  errors  can  not  be  sus¬ 
tained,  we  can  use  the  test  statistic  F*^  in  a  first-order  test  of  randomness. 
We  could,  for  example,  reject  the  hypothesis  of  randomness  if  F*^  >  10.  Such 
a  test  is  in  the  spirit  of  Saaty's  consistency  ratio  test  and  has  the  add¬ 
itional  advantage  of  being  scale-independent.  I  have  frequently  observed 
values  of  F*^  much  higher  than  10.  However,  F*^  depends  on  the  estimated 
weights.  Thus  a  comparisons  set  that  contains  high  values  for  comparisons 
will  have  a  high  F*^  value  provided  that  the  comparisons  are  fairly  consis¬ 
tent.  Hence  a  test  based  on  the  F  distribution  in  conjunction  with  the 
skewness  test  of  normality  is  preferable  to  using  the  F*^  test  statistic 
directly. 


Incomplete  Comparisons  Sets  and  Multiple  Comparisons 


25.  In  [3],  Crawford  and  Williams  state  that  the  LLS  method  is  easily 
extended  to  solve  more  general  problems.  One  such  problem  is  to  minimize 


SSE  .  S[(y..k  -  h Jk)‘l 


where  y.  =  In  a. 

Jijk  ljk 


and  *ijk  -jfjVijkr 


This  is  the  same  model  given  in  equation  3.  Except  in  the  special  case  where 
n^j  equals  one  for  all  i  and  j,  minimizing  equation  3  over  the  set  of  weights 
does  not  yield  a  closed-form  solution.  Using  the  method  of  steepest  descents 
[6]  easily  overcomes  this  problem.  This  method  iteratively  calculates  the 
weights  using 

b^  =  b^  +  zf^  1=1,..., n 
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where  b-^m^  is  the  estimate  of  (3-^  at  the  mtb  iteration, 

2  ■  -  bi  *  V<£i  -  £J>1  .1  <£1  -  £/’ 

J  J  1=1  J>1  J 

n  n.  . 
ij 

and  fj  =  2  E  I  (y-lk  "  bi  +  bl^* 
i=l  k=l 

It  is  convenient  to  initialize  these  calculations  by  setting  b^^  =  1/n  for 
all  1.  We  calculate  the  weights  iteratively  until  SSE  converges  within  pre¬ 
scribed  limits.  Thus,  in  contrast  to  the  EV  method,  it  is  easy  to  modify  the 
LLS  method  to  accomodate  comparisons  sets  that  are  incomplete  or  contain  mul¬ 
tiple  comparisons. 

26.  This  leads  to  an  additional  application  for  the  LLS  method.  If  the 
comparisons  set  is  incomplete,  the  LLS  method  can  be  used  to  estimate  the 
weights,  which  in  turn  can  be  used  to  estimate  the  missing  comparisons  of  an 
incomplete  comparisons  set  using  w^/w^,  allowing  use  of  the  EV  method.  (Of 
course,  weights  estimated  using  other  approaches  can  be  used  similarly.) 
However,  the  consistency  of  the  completed  SJM  will  reflect  in  part  the 
estimated  values  of  the  missing  comparisons.  If  the  approach  above  is  used, 
the  estimated  comparisons  that  complete  the  SJM  are  automatically  consistent, 
and  the  consistency  index  of  the  completed  SJM  may  indicate  a  high  degree  of 
consistency  in  the  matrix  as  a  whole.  This  tends  to  happen  especially  when 
the  SJM  is  sparse,  i.e.,  when  it  contains  relatively  few  comparisons  that  were 
not  estimated  from  weights.  Furthermore,  because  weights  calculated  using  the 
EV  and  LLS  methods  converge  as  an  SJM  approaches  perfect  consistency,  applying 
the  EV  method  to  a  sparse  SJM  that  has  been  completed  using  LLS  weights  would 
yield  a  set  of  weights  little  different  from  those  derived  using  the  LLS 
method.  Although  I  have  not  studied  this  matter  in  detail,  I  estimate  that 
the  EV  method  will  indicate  a  falsely  high  consistency  if  another  method  is 
used  to  estimate  comparisons  for  an  SJM  that  is  less  than  90%  complete. 
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27.  Because  the  LLS  method  is  not  matrix-based,  there  is  no  need  to  estimate 
missing  comparisons.  Furthermore,  the  approach  used  in  Appendix  A  to  develop 
a  scale-independent  test  of  consistency  remains  valid  with  only  minor  adjust¬ 
ments.  These  include  taking  the  sums  in  equations  3  and  10  only  over  the 
comparisons  that  have  been  made  and  adjusting  the  number  of  degrees  of  freedom 
in  calculating  MSE,  MSRU,  and  the  significance  or  risk  level.  Thus  the  test 
statistic  F*^  becomes 

F*y  =  MSRU'  /  MSE 


where 


MSRU'  =  SSRU'  /  n, 

SSRU'  =  S(y.jk2), 

MSE  =  SSE  /  \>2, 

n 

and  \>„  =  Z  Z  n.  .  -  n. 

2  i.i  3>iJ 

Under  the  hypothesis  of  randomness,  F*^  will  be  F-distributed  with  (n,  v2) 
degrees  of  freedom,  and  the  significance  and  risk  levels  for  the  hypothesis 
that  an  SJM  is  random  are  given  by  equations  8  and  9. 

28.  This  result  provides  valuable  insight  into  how  the  LLS  method's  ability 
to  detect  inconsistency  varies  with  the  number  of  comparisons.  Figure  2  shows 
the  relationship  between  the  risk  of  falsely  concluding  that  a  random  matrix 
is  non-random  (i.e.,  consistent)  and  the  number  of  comparisons.  The  figure 
shows  that,  for  a  12x12  SJM,  risk  is  largely  insensitive  to  both  the  value  of 
F*^  and  the  number  of  degrees  of  freedom  associated  with  SSE  once  the  latter 
value  exceeds  approximately  10.  Because  the  number  of  degrees  of  freedom  for 
SSE  is  the  number  of  SJM  entries  in  excess  of  that  required  to  form  a  ratio 
scale,  Figure  2  shows  that  little  further  ability  to  detect  inconsistency  is 
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igure  2.  Effect  of  Comparison  Set  Incompleteness  on  Ability  to  Detec 
Inconsistency  (12x12  matrix) 


obtained  if  there  are  more  than  n  +  V2  =  10  +  11  =  21  independent  comparisons. 
The  number  of  independent  comparisons  in  a  complete  12x12  SJM  is  66.  Thus  a 
12x12  SJM  that  is  one-third  complete  can  be  adequately  tested  for  consistency 
using  the  LLS  method.  (For  smaller  SJMs,  higher  fractions  of  completeness 
would  probably  be  required  to  test  consistency  adequately;  graphs  similar  to 
Figure  2  can  be  generated  for  these  SJMs.)  We  can  not  test  such  a  matrix 
using  the  consistency  ratio  approach  unless  ve  can  estimate  its  missing  com¬ 
parisons.  If  the  approach  suggested  in  paragraph  26  is  used,  the  EV  method 
will  provide  an  artificially  low  consistency  ratio  and  a  set  of  weights  little 
different  from  those  derived  using  the  LLS  method. 


Best  Fit  to  Most  of  the  Comparisons 

29.  As  with  all  least-squares  regression  methods,  the  LLS  method  is  sensitive 
to  outliers.  These  are  comparisons  whose  errors  are  large  relative  to  those 
of  other  comparisons  in  the  set.  Because  the  least-squares  regression  algo¬ 
rithm  chooses  weights  that  minimize  the  sum  of  squares  of  errors,  an  outlier 
causes  the  algorithm  to  choose  weights  that  do  not  best  represent  most  of  the 
comparisons.  An  alternative  method,  known  as  robust  regression  (RR),  select¬ 
ively  de-emphasizes  outliers. 


30.  Least-squares  regression's  sensitivity  to  outliers  arises  from  its  use  of 
the  square  of  an  error  as  the  penalty  for  that  error.  An  alternative  is  to 
use  a  penalty  that  is  less  sensitive  to  the  magnitude  of  the  error.  One  such 
penalty  function  is 


g(eijk)  =  1  -  exp  (-e. jk  ) 


As  Figure  3  shows,  this  function  resembles  an  inverted  normal  density  function 
centered  at  e. ..  =  0.  This  funct: 

1 J  K 

square  of  the  error  is  unbounded. 


centered  at  e. .,  =  0.  This  function  has  an  upper  limit  of  1,  whereas  the 

1 J  K 


31.  By  analogy  with  equation  3,  ve  can  define  a  robust  error  function  f'  that 
is  analogous  to  the  error  sum  of  squares.  This  function  is 
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Figure  3.  Comparison  of  Least-Squares  and  Robust  Regression  Penalty  Functions 


f'  =  S(g(e.jk)).  (11) 

To  minimize  the  robust  error  function  for  a  given  weight,  we  equate  the 
partial  derivative  of  f'  with  respect  to  the  weight  to  zero  and  solve.  This 
yields 


11  11  .  • 

ij 

E  E  (w}jk  e^jjc)  =  ®  1=1,..., n  (12) 

j=i  k-l 


2 

where  w^  =  exp  (-e-^  ).  The  factor  w.^k 

Welsh  weight  associated  with  y. ... 

i  J  *t 


is  frequently  referred  to  as  the 


32.  To  solve  equations  12,  we  expand  e^jjc  as  (y^j^  " 


n  n„  n 

bl  =  E  E  wljk  (yljk  +  b.)  /  E  E  w^  1=1 . n. 

j=l  k=l  j=l  k=l 


Unlike  equations  6,  each  of  these  equations  depends  on  weights  other  than  b^. 
We  solve  this  set  of  equations  using  an  iterative  technique.  For  the  m**1 
iteration  we  calculate 


(m) 


n 

=  E 

j 


w  (m-1> 
L  ljk 

1  k=l 


(y,  .,  +  b.(m  1))  / 
VJijk  j 


n  n^  . 

E  E  wljk(ffl'1)  1=1,..., n  (13) 

j=l  k=l 


where  the  superscript  (m)  indicates  that  the  factor  was  calculated  during  the 
m1*1  iteration.  Iteration  continues  until  f'^m^  _  converges  within 

prescribed  limits. 
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33.  As  with  other  iterative  techniques,  we  need  starting  values  for  the 

weights.  We  could  set  the  weights  equal,  e.g.,  =  1/n  for  all  1,  as  we 

did  in  the  LLS  method  for  incomplete  comparisons  sets.  However,  the  number  of 
iterations  required  for  f'  to  converge  depends  on  how  well  the  initial  values 
of  estimate  the  weights.  The  approach  taken  here  is  to  use  the  LLS  method 
to  provide  the  initial  estimates. 

34.  The  robust  regression  method  provides  weights  that  best  represent  most  of 
the  comparisons.  These  weights  are  based  on  all  the  comparisons  in  a  set 
rather  than  only  on  direct  comparisons,  as  in  the  LLS  method.  This  overcomes 
one  of  Saaty  and  Vargas'  objections  to  the  LLS  method  [7].  Furthermore,  be¬ 
cause  the  robust  regression  method  assigns  low  Welsh  weights  to  outliers, 
these  can  be  identified  by  examining  the  Welsh  weights.  However,  because  of 
the  nature  of  g(e..,),  we  cannot  partition  an  analogue  of  the  uncorrected 
total  sum  of  squares  into  error  and  regression  terms,  so  we  lose  the  ability 
to  perform  an  F-test  of  randomness.  If  the  RR  method  does  not  significantly 
modify  the  LLS  weights,  an  F-test  based  on  the  LLS  weights  should  remain 
valid.  This  will  occur  if  the  comparisons  set  contains  no  outliers.  Such  a 
situation  exists  if  f'^,  the  value  of  equation  11  calculated  using  LLS 
weights,  is  nearly  the  same  as  the  final  value  of  f'  '  calculated  using  the 
weights  given  by  equations  13. 

Summary  of  Extensions 

35.  Table  3  compares  the  EV  method  with  the  enhanced  LLS  and  RR  methods.  The 
EV  method's  advantages  and  limitations  are  nearly  the  same  as  those  shown  in 
Table  1.  The  only  change  is  that  the  eigenvalue  method  can  be  adapted  to 
SJMs  that  are  up  to  perhaps  10%  incomplete,  given  another  method  for  estimat¬ 
ing  the  weights.  The  LLS  method  is  one  such  method.  However,  if  the  SJM  is 
sparsely  populated  with  comparisons,  and  missing  comparisons  are  estimated 
using  Wj/Wj,  the  SJM  will  appear  to  be  almost  perfectly  consistent  and  the 
weights  calculated  by  the  EV  method  will  be  only  slightly  different  than  those 
calculated  by  the  non-EV  method  used  to  estimate  the  missing  comparisons. 
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TABLE  3.  COMPARISON  OF  EIGENVALUE,  ENHANCED  LOGARITHMIC 
LEAST-SQUARES,  AND  ROBUST  REGRESSION  METHODS 


EV  Method 

Enhanced  LLS  Method 

RR  Method 

Advantages 

Ratio  scale  depends  on 

all  comparisons 

Works  with  incomplete 

comparisons  sets, 

within  limits 

Works  with  comparisons 

sets  that  are  incom¬ 
plete  or  contain 

multiple  comparisons 

Consistency  metric  is 

scale-independent 

Works  with  comparisons 

sets  that  are  incom¬ 
plete  or  contain 

multiple  comparisons 

Ratio  scale  depends  on 

all  comparisons 

Insensitive  to  outliers 

Limitations 

Consistency  metric  is 

scale-dependent 

Consistency  ratio  test 

Consistency  test  is 
distribution-dependent 

Sensitive  to  outliers 

No  consistency  metric 

is  not  uniform  for 
SJMs  of  different 
orders 

Does  not  work  with  com¬ 
parisons  sets  that 
contain  multiple 
comparisons 
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36.  The  enhancements  discussed  above  eliminate  several  of  the  LLS  method's 
limitations  without  significantly  degrading  its  advantages.  There  is  now  a 
scale-independent  consistency  metric  for  both  complete  and  incomplete  com¬ 
parisons  sets.  Furthermore,  the  relationship  between  risk  and  the  number  of 
comparisons  provides  insight  into  the  number  of  comparisons  needed  to  detect 
inconsistency.  This  consistency  metric  can  be  used  to  perform  tests  of  con¬ 
sistency  that  are  uniform  for  comparisons  sets  of  different  sizes. 

37.  In  addition,  the  robust  regression  technique  overcomes  the  LLS  method's 
sensitivity  to  outliers  by  selectively  de-emphasizing  the  comparisons  that  are 
most  inconsistent,  thereby  providing  a  set  of  weights  that  best  represent  most 
of  the  comparisons. 

EXAMPLES 


Example  1 

38.  Table  4  shows  an  SJM  that  contains  known  inconsistencies.  During  data 
collection,  the  comparisons  involving  entities  F,  G,  H,  and  I  were  recorded 
incorrectly,  i.e.,  the  reciprocals  of  the  correct  comparisons  were  recorded. 
(I  have  found  that  recording  errors  of  this  kind  are  fairly  common.)  Table  5 
presents  selected  statistics  for  this  matrix,  including  the  EV,  LLS,  and 
robust  regression  weights  for  the  entities.  Because  the  matrix  contains  only 
a  single  comparison  greater  than  9,  using  the  random  index  for  Saaty's 
nine-value  scale  is  reasonable.  The  consistency  ratio  is  0.72,  much  higher 
than  required  to  accept  the  hypothesis  of  randomness  using  Saaty's  test. 
However,  Saaty's  risk  of  falsely  rejecting  a  random  matrix  with  this  consis¬ 
tency  ratio  is  0.038.  This  shows  that  a  random  SJM  of  the  same  order  as  the 
SJM  in  Table  4  is  unlikely  to  have  a  consistency  ratio  less  than  0.72. 
Similarly,  the  F*^  value  for  the  SJM  is  2.65  and  the  associated  risk  of  false 
rejection,  based  on  the  F  test,  is  0.024.  Table  5  also  shows  a  significant 
difference  between  the  LLS  and  robust  regression  weights,  especially  for  the 
entities  involved  in  incorrectly  recorded  comparisons.  Comparing  f'^  and 
f'^m^  confirms  this  difference,  indicating  that  the  robust  regression  method 
substantially  revised  the  LLS  values  used  to  initialize  it.  Table  6,  the 


23 


TABLE  4.  SUBJECTIVE  JUDGMENT  MATRIX  FOR  EXAMPLE  1 
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TABLE  5.  SELECTED  STATISTICS  FOR  EXAMPLE  1 


Original  Matrix 


Corrected  Matrix 


EV 

LLS 

RR 

EV 

LLS 

RR 

Consistency 

0.72 

2.65 

— 

0.11 

22.64 

— 

Test  Statistic* 

Risk** 

0.038 

0.024 

— 

0 

0 

— 

Skewness 

_ 

0.089 

— 

_ 

0.468 

_ 

Probabili ty 

Robust  Error 

_ 

32.28 

20.08 

— 

15.36 

15.07 

(f') 

Entity 

— 

— 

—  Ratio 

Scales  - 

— 

— 

A 

0.073 

0.064 

0.046 

0.052 

0.053 

0.049 

B 

0.098 

0.107 

0.085 

0.080 

0.089 

0.090 

C 

0.144 

0.180 

0.129 

0.152 

0.150 

0.139 

D 

0.231 

0.300 

0.236 

0.245 

0.251 

0.251 

E 

0.031 

0.031 

0.023 

0.025 

0.026 

0.026 

F 

0.032 

0.042 

0.078 

0.083 

0.083 

0.090 

G 

0.162 

0.143 

0.358 

0.318 

0.301 

0.308 

H 

0.129 

0.093 

0.033 

0.030 

0.032 

0.034 

I 

0.100 

0.042 

0.011 

0.014 

0.014 

0.013 

*  Consistency  ratio  for  EV,  F*^  for  LLS 

**  Probability  of  falsely  rejecting  hypothesis  of  randomness.  Saaty's  risk 
for  EV  method,  based  on  F*^  for  LLS 
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TABLE  6.  WELSH  WEIGHTS  FOR  EXAMPLE  1 
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table  of  Welsh  weights  for  this  SJM,  shows  that  very  low  Welsh  weights  were 
assigned  to  comparisons  between  entities  F,  G,  H,  and  I,  clearly  showing  that 
comparisons  involving  these  entities  are  outliers. 

39.  Once  this  inconsistency  was  detected,  the  respondent  who  provided  the 
comparisons  was  consulted.  He  corrected  the  SJM  by  changing  all  outlier 
comparisons  to  their  reciprocals.  Table  5  also  presents  statistics  for  the 
corrected  SJM.  The  corrections  bring  the  SJM  to  the  threshold  of  passing 
Saaty's  consistency  ratio  test.  However,  Saaty's  risk  for  the  corrected 
matrix  is  nil.  There  is  a  dramatic  increase  in  the  value  of  F*y,  and  the  risk 
of  false  rejection  based  on  the  F  test  is  nil.  The  LLS  and  robust  regression 
weights  for  the  corrected  matrix  are  only  slightly  different,  as  are  the 
values  of  f'^  ^  and  f'^m\  suggesting  that  there  are  no  additional  outliers. 

40.  Also  note  in  Table  5  that  the  RR  ratio  scale  for  the  original  SJM  is  much 
closer  to  the  EV  and  LLS  scales  for  the  corrected  matrix  than  these  scales  for 
the  original  one.  Although  correcting  the  outlier  comparisons  has  a  signif¬ 
icant  effect  on  the  EV  and  LLS  ratio  scales,  this  effect  is  only  slightly 
larger  than  that  of  the  RR  method  de-emphasizing  the  outliers  in  the  original. 
In  addition,  the  RR  ratio  scales  for  the  original  and  corrected  matrices  are 
nearly  the  same.  This  suggests  that,  when  a  comparisons  set  is  inconsistent 
and  the  respondent  cannot  be  contacted  in  a  timely  manner  to  review  his 
comparisons,  the  RR  ratio  scale  should  be  used. 

Example  2 

41.  Table  7  shows  an  incomplete  SJM  Smith  [8]  used  to  illustrate  an  approach 
to  completing  an  incomplete  SJM  and  resolving  inconsistencies.  In  Table  7, 
zeroes  indicate  missing  comparisons.  The  matrix  is  sparse,  i.e.,  it  contains 
only  one-third  of  the  comparisons  in  a  complete  SJM.  Furthermore,  Smith's 
matrix  is  12x12,  the  order  for  which  Figure  2  was  generated.  As  shown  there, 
we  obtain  little  additional  ability  to  detect  inconsistency  by  completing  more 
than  one-third  of  a  12x12  matrix. 
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TABLE  7.  SUBJECTIVE  COMPARISON  MATRIX  FOR  EXAMPLE  2 
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42.  Table  8  presents  selected  statistics  for  Smith's  matrix.  As  the  table 
shows,  this  SJM  is  inconsistent,  and  rejecting  the  hypothesis  of  randomness 
entails  a  high  risk.  Because  the  matrix  was  designed  to  exemplify  an  approach 
to  resolving  inconsistency,  this  result  is  not  surprising.  The  LLS  and  robust 
regression  methods  produce  ratio  scales  that  accurately  represent  the  incom¬ 
plete  SJM.  In  this  case,  the  consistency  ratio  of  the  SJM  completed  using  LLS 
weight  estimates  is  0.06,  sufficiently  low  to  pass  Saaty's  consistency  test. 
However,  this  consistency  ratio  is  artificially  low  because  the  estimated  com¬ 
parisons  that  comprise  two-thirds  of  the  matrix  are  automatically  consistent. 
Hence  Saaty's  test  does  not  provide  an  adequate  basis  for  testing  consistency 
in  this  case. 

SUMMARY 


43.  This  technical  memorandum  has  presented  two  extensions  of  the  LLS  method 
for  analyzing  subjective  comparisons.  The  first  of  these  is  a  scale- 
independent  test  for  the  consistency  of  a  comparisons  set.  This  test,  based 
on  analysis-of-variance  techniques,  uses  a  test  statistic  that  is  F- 
distributed  if  the  error  terms  are  log-normal-distributed  and  the  comparisons 
are  random.  It  is  easy  to  apply  this  test  to  comparisons  sets  that  are  incom¬ 
plete  or  contain  multiple  comparisons  between  pairs  of  entities.  The  risk  of 
falsely  rejecting  the  hypothesis  of  randomness  based  on  this  test  statistic 
also  provides  valuable  insights  into  the  trade-off  between  comparisons  set 
completeness  and  our  ability  to  detect  inconsistency. 

44.  The  second  extension  was  using  robust  regression  to  find  a  ratio  scale 
that  best  represents  most  of  the  comparisons  in  a  set.  Because  the  LLS  method 
is  a  least-squares  method,  its  results  are  sensitive  to  outliers.  The  robust 
regression  method  selectively  de-emphasizes  outliers.  This  method  also  allows 
comparisons  that  do  not  directly  involve  an  entity  to  affect  its  weight,  over¬ 
coming  one  of  Saaty  and  Vargas'  objections  to  the  LLS  method. 

45.  These  enhancements  to  the  LLS  method  eliminate  many  of  the  limitations  of 
the  basic  LLS  method  without  introducing  substantial  new  limitations.  Along 
with  the  previously-known  advantages  of  the  LLS  method,  these  extensions 
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TABLE  8.  SELECTED  STATISTICS  FOR  EXAMPLE  2 


EV 

LLS 

RR 

Consistency 

0.06 

0.82 

_ 

Test  Statistic* 

Risk** 

0 

0.635 

— 

Skewness 

_ 

0.431 

_ 

Probability 

Robust  Error 

—  — 

11.48 

8.20 

<f') 

Entity  - 

— 

-  Ratio  Scale  - 

A 

0.163 

0.140 

0.130 

B 

0.173 

0.169 

0.074 

C 

0.055 

0.057 

0.058 

D 

0.062 

0.062 

0.122 

E 

0.038 

0.042 

0.044 

F 

0.071 

0.078 

0.073 

G 

0.067 

0.073 

0.077 

H 

0.043 

0.040 

0.113 

I 

0.133 

0.139 

0.091 

J 

0.047 

0.047 

0.060 

K 

0.081 

0.088 

0.084 

L 

0.066 

0.066 

0.075 

*  Consistency  ratio  for  EV,  F*^  for  LLS 

**  Probability  of  falsely  rejecting  hypothesis  of  randomness.  Saaty's  risk 
for  EV  method,  based  on  F*y  for  LLS 


30 


contribute  to  making  the  logarithmic  least-squares  method  an  attractive 
alternative  to  Saaty's  eigenvalue  method  for  analyzing  subjective  comparisons. 
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APPENDIX  A 


Statistical  Test  for  Random  Comparisons 

A. 1  This  appendix  presents  a  statistical  test  for  whether  the  entries  in 
a  subjective  comparisons  set  are  random.  The  test  is  based  on  the  logarithmic 
least-squares  (LLS)  method  for  estimating  the  relative  values  or  weights  of 
the  entities  represented  in  the  comparisons  set.  As  Crawford  and  Williams  [3] 
showed,  the  logarithmic  least-squares  method  is  equivalent  to  log-linear 
regression,  allowing  us  to  use  analysis-of-variance  techniques  to  formulate  a 
test  of  randomness  for  comparisons  sets.  However,  because  of  a  special  char¬ 
acteristic  of  the  log-linear  model  used  to  fit  the  comparisons  set,  the  usual 
analysis-of-variance  approach  to  testing  randomness  is  inappropriate.  This 
appendix  develops  an  alternative  approach,  also  based  on  the  analysis  of  var¬ 
iance,  that  tests  randomness  and  provides  intuitively  appealing  metrics  for 
the  consistency  of  a  comparisons  set.  It  develops  this  approach  for  a 
(complete)  subjective  judgment  matrix  (SJM),  but  it  is  easy  to  generalize  the 
result  to  comparisons  sets  that  are  incomplete  or  contain  multiple  compar¬ 
isons. 


A. 2  An  SJM  is  a  positive  reciprocal  matrix  whose  entries  are  subjective 
estimates  of  the  relative  importance  of  entities  i  and  j.  A  positive 
reciprocal  matrix  is  one  whose  entries  are  all  positive  and  satisfy  a^  = 

1/a^j  for  all  i  and  j.  The  entries  in  an  SJM  are  usually  viewed  as  estimates 
of  x^/Xj,  where  x^.  is  the  weight  of  entity  i.  In  addition  to  estimating  the 
values  of  the  weights,  we  are  concerned  with  the  consistency  of  the  SJM.  If  A 
=  is  an  SJM,  A  is  perfectly  consistent  if  a^  =  a^a^j  f°r  all  i,  j,  and 

k.  We  want  to  test  the  hypothesis  that  the  a^  are  random.  If  they  are,  then 
A  is  not  consistent.  This  is  a  hypothesis  we  would  like  to  reject. 


A. 3  Crawford  and  Williams  [3]  showed  that  the  logarithmic  least-squares 
(LLS)  method  of  estimating  the  ratio  scale  underlying  an  SJM  minimizes 


m(A,C)  =  S((ln  a.j 


In  w.  +  In  w.)  ) 
i  J 


(A.l) 


A-l 


where 


S(z . .)  = 
ij 


n 

£ 


Z  z , 


i=l  j>i 


ij 


and  m(A,C)  is  the  sum  of  squares  of  logarithmic  errors  between  A  and  the 
nearest  consistent  matrix  C.  The  metric  m(A,C)  will  be  referred  to  below  as 
the  m-distance  between  matrices  A  and  C,  and  the  consistent  matrix  having 
the  smallest  m-distance  to  A  is  the  m-closest  consistent  matrix.  C  is  a 
consistent  matrix  whose  entries  c^  equal  w./v.,  and  w^  is  the  weight  assoc¬ 
iated  with  entity  i.  This  method  gives 


w. 

1 


( 


n  a.  . 
1-1 1J 


.  1/n 


i=l, . . .  ,n 


(A. 2) 


where  c  is  an  arbitrary  scaling  constant.  The  weight  of  entity  i  is  the 

t  h 

geometric  mean  of  the  entries  in  the  l  row  of  A.  (For  this  reason,  the 
LLS  method  is  frequently  referred  to  as  the  geometric  mean  (GM)  method.) 
Crawford  and  Williams  [3]  show  that  using  equation  A. 2  to  calculate  weights 
is  equivalent  to  doing  a  log-linear  regression  on  the  entries  of  A.  This 
regression  has  the  form 


ij 


n 

=  E 
k 


fjViJk  + 


Eij 


(A. 3) 


where  y...  =  In  a^, 

\  =  ln  V 


and 


^ijk=  t*ie  k*  th  independent  variable  for  the  (i,j)  entry  of  A, 


eij 


error  term. 


X. .,  equals  1  if  k  =  i  *  j ,  -1  if  k  =  j  t  i,  and  zero  otherwise.  Equation 
l  jk 

A.l  is  the  sum  of  squared  errors  for  the  log-linear  regression,  i.e., 

2 

m(A,C)  =  S(ejj  ),  where  e^  is  an  estimate  of  j . 
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A. 4  One  noteworthy  characteristic  of  equation  A. 3  is  that  it  does  not 

contain  a  constant  term,  i.e.,  there  is  no  k  for  which  X...  =1  for  all  i 

ljk 

and  j.  In  a  general  regression  model,  there  is  usually  a  constant  term  that 
is  fitted  in  the  same  manner  as  the  other  regression  coefficients.  This 
constant  term  represents  the  estimated  value  of  y  when  all  of  the  values 
equal  zero,  i.e.,  the  y-intercept.  In  such  a  "fitted-intercept"  model, 
we're  assured  that  the  error  terms  sum  to  zero.  Suppose  the  model  is 


*  - 


(A. 4) 


where  X.A  =  1  for  all  i.  Then 
lO 


2 

SSE  =  Z  e.  (A. 5) 

i  =  l 


and 


n 

3SSE/3bn  =21  e. [3e./3bn] 
0  .  ,  l1  i  0J 

x=l 


where  b^  is  the  least-squares  estimator  of  3q.  The  residual  errors  e^  equal 

y .  -  y . ,  where 
J  i  J  i 

P-1 


so  3e./3b0  =  -X.0  =  -1 
and 
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n 

3SSE/3b0  =  -2  Z  e.. 

i=l 


But  3SSE/3bQ  =  0  is  a  necessary  condition  for  minimizing  equation  A. 5.  It 
follows  that 


n 

r 

i=l 


ei 


=  0 


is  a  necessary  condition  for  minimizing  SSE  in  the  fitted-intercept  model 
given  by  equation  A. 4.  In  the  zero-intercept  modelof  equation  A. 3,  there  is 
no  constant  term  and  the  sum  of  errors  is  unconstrained.  As  will  be  shown 
below,  this  has  a  significant  impact  on  designing  a  scale-free  test  of 
randomness. 

A. 5  In  regression  theory,  an  F  test  is  frequently  used  to  assess  whether 
there  is  any  meaningful  relationship  between  the  dependent  and  independent 
variables.  If  there  is  no  relationship  between  the  variables,  a  regression 
of  the  form 

v.  =  y  +  e . , 

where  y  is  the  average  of  the  y^'s,  suffices.  This  model  is  the  same  as 
that  of  equation  A. 4,  but  with  b^  =  0,  k=l,...,p-l.  The  F  test  addresses 
the  hypothesis  that  the  y^'s  are  random  variations  about  a  constant  mean. 

An  F  test  examines  this  hypothesis  using  the  criterion  that  b^  ■  0,  k=l,..., 
p-1.  This  test  uses  an  analysis-of-variance  approach  to  subdivide  the  var¬ 
iance  of  the  y^'s  into  separate  variances  that  represent  the  variance  of  the 
dependent  variables  around  the  regression  line  and  the  variance  of  the 
regression  line  around  the  mean  value.  Mathematically,  for  the  model  given 
by  equation  A. 4, 

SSTO  =  SSE  +  SSR  (A. 6) 
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where  SSTO  is  the  total  sum  of  squares  (i.e.,  the  variance  of  the  y^'s),  SSE 
is  error  sum  of  squares,  and  SSR  is  the  regression  sum  of  squares.  SSTO, 
SSE,  and  SSR  are  given  by 


SSTO  =  Z  (y.  -  y)2, 
i=l 

n  2 
SSE  =  I  e .  , 

i=l  1 

n  p-i  9 

and  SSR  =  Z  Z  (b,X.,  -  y)  , 
i=l  k=0  K 

where  n  is  the  number  of  independent  y  values. 

A. 6  For  the  fitted-intercept  model,  SSE  and  SSR  are  used  to  calculate  a 
test  statistic  F*  given  by 

F*  =  MSR  /  MSE  (A. 7) 

where  MSR  and  MSE  are  the  regression  and  error  mean  squares,  respectively. 
They  are  defined  as 

MSR  =  SSR  /  (p-1) 


and 


MSE  =  SSE  /  (n-p). 

If,  as  is  usually  assumed,  the  error  terms  in  equation  A. 4  are  normally 

2  2 

distributed  with  mean  0  and  variance  a  ,  MSR  and  MSE  are  X  -distributed  with 
p-1  and  n-p  degrees  of  freedom,  respectively.  The  expected  value  of  MSE  is 
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.  The  ratio  of  two  independent  X^-distributed  random  variables  is 

F-distributed.  Furthermore,  if  b,  =  0,  k=l,...,p-l,  the  expected  value  of 
2  K 

MSR  is  a  ,  and  F*'s  expected  value  is  1.  Hence  the  significance  level  for 
the  hypothesis  that  the  y  values  are  random,  i.e.,  the  probability  of 
concluding  correctly  that  the  y^'s  are  random,  using  the  criterion  that 
b^  =  0,  k=l, . . . ,p-l,  is 

F* 

s  =  /  f(x;  p-1,  n-p)  dx 
0 

where  s  is  the  significance  level  and  f(x;  p-1,  n-p)  is  the  density  function 
of  the  F  distribution  with  (p-1,  n-p)  degrees  of  freedom.  Because  the  value 
of  s  increases  as  F*  increases,  the  hypothesis  of  randomness  is  increasingly 
supported  by  increasing  values  of  F*.  Conversely,  the  probability  of 
falsely  rejecting  the  hypothesis  that  the  y^'s  are  random  is 

a  =  1  -  s 

where  a  is  frequently  referred  to  as  the  risk  associated  with  the  hypothesis 
of  randomness.  This  risk  decreases  as  F*  increases. 

A. 7  In  their  derivation  of  equation  A. 6,  Neter  and  Vasserman  [4]  use  the 
fact  that,  in  the  fitted-intercept  model,  the  error  terms  sum  to  zero.  In 
the  zero-intercept  model  of  equation  A. 3,  the  error  terms  do  not  generally 
sum  to  zero.  Defining  SSE  as  in  equation  A. 5,  we  have 

n  p 

aSSE/Sb,  =  -2  E  (e.  I  X..  abJc/3b1).  (A. 8) 

i=i  k=IK 

Because  db^/ab-^  =  1  for  k  =  1,  1=1,... p,  and  zero  otherwise,  equation  A. 8 
reduces  to 


assE/at^  =  -2 


n 


Zi!iXil 


1  *  1 > • • • t  P •  (A. 9 ) 


A  necessary  condition  for  minimizing  SSE  is  that  the  p  equations  given  by 
equation  A. 9  all  equal  zero.  However,  for  the  zero-intercept  model,  there 
is  no  value  of  1  for  which  equals  1  for  all  i  and  j.  Accordingly,  the 
error  terms  need  not  sum  to  zero  as  a  necessary  condition  for  minimizing  SSE 
in  the  zero-intercept  model. 

A. 8  In  the  zero-intercept  case,  it  can  be  shown  that,  using  the  notation  of 
equation  A. 6, 

SSTO  =  SSE  -  2yS(e.j)  +  SSR  (A. 10) 

Hence,  unlike  the  fitted-intercept  case,  the  total  sum  of  squares  cannot  be 
separated  into  error  and  regression  sums  of  squares.  Accordingly,  F*,  the 
F-test  statistic  given  by  equation  A. 7,  does  not  completely  represent  the 
total  sum  of  squares.  Although  it  is  possible  to  formulate  a  test  statistic 
similar  to  F*  based  on  equation  A. 10,  it  will  not  be  F-distributed.  How¬ 
ever,  if  the  error  terms  have  mean  values  of  zero,  S(e„)  will  usually  be 
small  and  the  test  statistic  will  be  approximately  F-distributed. 

A. 9  Alternatively,  we  can  use  a  test  based  on  partitioning  the  uncorrected 
total  sum  of  squares  (SSTOU).  SSTOU  is  given  by 

SSTOU  =  S(yij2)  =  m(A,U) 

where  U  is  the  unit  matrix,  i.e.,  a  matrix  all  of  whose  entries  equal  1.  It 
is  easy  to  show  that 

SSTOU  =  SSE  +  SSRU  (A. 11) 


where  SSRU  is  the  uncorrected  regression  sum  of  squares  given  by 


A-7 


SSRU  =  S(y.j) 


and 


ij 


n 

=  Z 
k=l 


b,  X.  ..  . 
k  ljk 


Furthermore,  both  SSE  and  SSRU  have  intuitively  appealing  interpretations. 
SSE  is  the  m-distance  between  A  and  the  m-closest  consistent  matrix  C,  whose 
entries  c.^  =  w^/w.. ;  SSRU  is  the  m-distance  between  C  and  U.  Thus  equation 
A. 11  can  be  rewritten  as 

m(A, U)  =  m(A,C)  +  m(C,U). 

If  the  entries  of  A  are  random,  we  expect  its  m-closest  consistent  matrix  to 
be  nearly  a  unit  matrix,  and  SSRU  =  m(C,U)  will  be  small. 

A. 10  Under  the  assumption  that  errors  are  normally  distributed,  SSE  and 
SSRU  are  X  -distributed  with  N-n  and  n  degrees  of  freedom,  respectively. 

Here  N  =  n(n-l)/2,  the  number  of  independent  entries  in  the  SJM.  The  mean 
squares  associated  with  SSE  and  SSRU  are 

MSE  =  SSE  /  (N-n) 


and 


MSRU  =  SSRU  /  n. 

The  expected  values  of  the  mean  squares  are 
E(MSE)  *  c2 


2  2 

E(MSRU)  =  <r  +  (n-1 )  Z  &  /  n  -  I  &  g, 

1.  *1  K  ^ 


and 


where  3.  is  the  expected  value  of  b..  Under  the  hypothesis  of  randomness, 
i  1  2 

3^  equals  zero  for  all  k  and  E(MSRU)  equals  a  .  This  allows  us  to  formulate 

an  uncorrected  F  test  statistic 


F*(J  =  MSRU  /  MSE 

that  is  F-distributed  with  (n,  N-n)  degrees  of  freedom.  The  significance 
level  associated  with  the  hypothesis  of  randomness  is 

F* 

U 

s  =  /  f(x;  n,  N-n)  dx  (A. 12) 

0 

and  the  risk  of  falsely  rejecting  the  hypothesis  is 

a  =  1  -  s.  (A. 13) 

A. 11  These  results  are  readily  extended  to  comparisons  sets  that  are 
incomplete  or  contain  multiple  comparisons  between  entities.  All  that  is 
required  is  to  take  sums  over  all  comparisons.  Thus  the  summation  operator 
defined  in  paragraph  A. 3  becomes 


S(z 


ijl 


) 


li  li  .  . 

ij 

=  Z  Z  Z 
i=l  j>i  1=1 


where  n^  is  the  number  of  comparisons  between  entities  i  and  j.  The  number 
of  degrees  of  freedom  associated  with  SSE  is  N-n,  where  N  is  redefined  as 


n 

N  =  Z 
i=l 


Z  n.  .. 
j>i  ^ 


For  the  special  case  of  a  complete  SJM,  N  equals  n(n-l)/2.  The  significance 
and  risk  levels  are  given  by  equations  A. 12  and  A. 13. 
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APPENDIX  B 


Tables  for  Tests  of  Error  Normality  and  Independence 


B.l  This  appendix  presents  tabulated  data  that  can  be  used  to  test  the  hy¬ 
pothesis  that  errors  in  complete  subjective  judgment  matrices  (SJMs)  are 
independent  and  identically  distributed.  The  tables  can  also  be  used  to  test 
this  hypothesis  for  comparisons  sets  that  are  incomplete  or  contain  multiple 
comparsions  between  entities. 

B.2  Tables  B.l  through  B.20  present  values  of  six  statistics  for  SJMs  of 
order  3  through  22.  The  statistics  are: 

XBAR  -  the  mean  error; 

2 

VAR  -  the  variance  a  of  the  errors; 

| GAMMA 1 |  -  the  absolute  value  of  error  skewness,  i.e., 

| GAMMA 1 |  =  |  u3  /  a3  | 

N  3 

where  u,  =  E  (X.-XBAR)J)  /  (N-l), 
i-1  1 

N  =  n  (n-l)  /  2, 
and  n  is  the  order  of  the  SJM; 

GAMMA2  -  the  kurtosis  of  errors,  i.e., 

GAMMA2  =  (  y4  /  c4  )  -  3 

N  4 

where  y,  =  (E  (X.-XBAR)  )  /  (N-l); 
i=l  1 

CHISQC  -  a  X2  statistic  that  compares  the  number  of  positive  and 
negative  error  terms; 

and  PC2C  -  the  probability  that  CHISQC  is  X^-distributed  with  one  degree 
of  freedom. 

Each  table  gives  the  mean  and  variance  of  each  statistic  as  well  as  selected 
percentage  points. 
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Table  B.l:  Statistics  for  3x3  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

Mean 

0.00815 

1.08184 

0.37452 

-2.00000 

1.00000 

Var. 

0.34821 

1.25151 

0.03058 

0.00000 

1.33454 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-1.80137 

0.00081 

0.00010 

-2.00127 

0.33333 

1.00 

-1.36577 

0.01233 

0.00766 

-2.00017 

0.33333 

5.00 

-0.97805 

0.05438 

0.05036 

-2.00000 

0.33333 

10.00 

-0.76288 

0.11348 

0.09974 

-2.00000 

0.33333 

15.00 

-0.61534 

0.19208 

0.14244 

-2.00000 

0.33333 

20.00 

-0.48616 

0.25596 

0.19106 

-2.00000 

0.33333 

25.00 

-0.38749 

0.30064 

0.23804 

-2.00000 

0.33333 

30.00 

-0.29612 

0.36002 

0.27542 

-2.00000 

0.33333 

35.00 

-0.21624 

0.44432 

0.31905 

-2.00000 

0.33333 

40.00 

-0.13248 

0.52956 

0.35090 

-2.00000 

0.33333 

45.00 

-0.06626 

0.61857 

0.38170 

-2.00000 

0.33333 

50.00 

-0.00089 

0.70627 

0.41762 

-2.00000 

0.33333 

55.00 

0.09669 

0.82845 

0.44924 

-2.00000 

0.33333 

60.00 

0.17998 

0.93104 

0.47111 

-2.00000 

0.33333 

65.00 

0.26256 

1.05952 

0.49540 

-2.00000 

0.33333 

70.00 

0.33256 

1.26693 

0.51749 

-2.00000 

0.33333 

75.00 

0.41378 

1.48020 

0.53452 

-2.00000 

0.33333 

80.00 

0.51155 

1.72608 

0.55133 

-2.00000 

3.00000 

85.00 

0.62798 

2.03814 

0.56248 

-2.00000 

3.00000 

90.00 

0.74359 

2.59913 

0.56946 

-2.00000 

3.00000 

95.00 

0.96504 

3.34821 

0.57553 

-1.99999 

3.00000 

99.00 

1.29685 

5.08265 

0.57717 

-1.99979 

3.00000 

.00.00 

1.97259 

7.95851 

0.57735 

-1.99755 

3.00000 

PC2C 

0.55641 

0.04332 


PC2C 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.43630 

0.91674 

0.91674 

0.91674 

0.91674 

0.91674 

0.91674 
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Table  B.2:  Statistics  for  4x4  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

Mean 

0.00775 

1.02735 

0.45069 

-1.21155 

0.98364 

Var. 

0.17240 

0.42520 

0.11442 

0.24271 

1.73584 

Percentage  Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-1.50905 

0.05281 

0.00091 

-2.12499 

0.00000 

1.00 

-1.03679 

0.11829 

0.00552 

-2.03941 

0.00000 

5.00 

-0.68521 

0.22560 

0.03464 

-1.85245 

0.00000 

10.00 

-0.53083 

0.32342 

0.06739 

-1.75170 

0.00000 

15.00 

-0.43037 

0.41961 

0.10592 

-1.69707 

0.00000 

20.00 

-0.33919 

0.49794 

0.14171 

-1.65787 

0.00000 

25.00 

-0.27161 

0.56509 

0.17705 

-1.59373 

0.00000 

30.00 

-0.19908 

0.62402 

0.22012 

-1.54824 

0.00000 

35.00 

-0.14042 

0.67940 

0.26429 

-1.48396 

0.66667 

40.00 

-0.08046 

0.75092 

0.29992 

-1.42215 

0.66667 

45.00 

-0.03157 

0.81667 

0.33557 

-1.35539 

0.66667 

50.00 

0.01517 

0.89961 

0.38237 

-1.28935 

0.66667 

55.00 

0.08020 

0.96531 

0.41940 

-1.23223 

0.66667 

60.00 

0.12683 

1.05688 

0.47153 

-1.16751 

0.66667 

65.00 

0.17451 

1.14153 

0.53010 

-1.08139 

0.66667 

70.00 

0.22871 

1.24833 

0.58578 

-0.98463 

0.66667 

75.00 

0.27797 

1.33452 

0.64333 

-0.89837 

0.66667 

80.00 

0.35003 

1.48134 

0.72162 

-0.79006 

2.66667 

85.00 

0.44184 

1.64925 

0.81187 

-0.67169 

2.66667 

90.00 

0.52263 

1.89103 

0.95534 

-0.52925 

2.66667 

95.00 

0.68186 

2.28010 

1.13038 

-0.25083 

2.66667 

99.00 

0.94866 

2.97527 

1.40082 

0.13967 

6.00000 

00.00 

1.45058 

5.07716 

1.61022 

0.46403 

6.00000 

PC2C 

0.45329 

0.12547 


PC2C 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.58579 

0.58579 

0.58579 

0.58579 

0.58579 

0.58579 

0.58579 

0.58579 

0.58579 

0.89751 

0.89751 

0.89751 

0.89751 

0.98571 

0.98571 
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Table  B.3:  Statistics  for  5x5  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

PC2C 

Mean 

-0.00331 

0.97609 

0.42357 

-0.77916 

0.99964 

0.47790 

Var. 

0.09662 

0.20302 

0.11757 

0.45613 

1.85987 

0.10668 

Percentage  Points  of  Statistics 


X 

XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

PC2C 

0.09 

-1.02221 

0.12505 

0.00111 

-1.81560 

0.00000 

0.00000 

1.00 

-0.75197 

0.23622 

0.01011 

-1.74502 

0.00000 

0.00000 

5.00 

-0.51788 

0.38264 

0.03455 

-1.56581 

0.00000 

0.00000 

10.00 

-0.40477 

0.47720 

0.06001 

-1.46896 

0.00000 

0.00000 

15.00 

-0.32945 

0.53710 

0.08936 

-1.41197 

0.00000 

0.00000 

20.00 

-0.25809 

0.59528 

0.11615 

-1.33927 

0.00000 

0.00000 

25.00 

-0.20925 

0.64496 

0.14953 

-1.27283 

0.00000 

0.00000 

30.00 

-0.16082 

0.69551 

0.17970 

-1.20239 

0.40000 

0.47291 

35.00 

-0.11585 

0.74166 

0.22062 

-1.14119 

0.40000 

0.47291 

40.00 

-0.07877 

0.79236 

0.25702 

-1.07502 

0.40000 

0.47291 

45.00 

-0.04716 

0.85213 

0.30441 

-1.00986 

0.40000 

0.47291 

50.00 

-0.00013 

0.90199 

0.33467 

-0.94066 

0.40000 

0.47291 

55.00 

0.03628 

0.95633 

0.38614 

-0.86284 

0.40000 

0.47291 

60.00 

0.06701 

1.01378 

0.43509 

-0.76880 

0.40000 

0.47291 

65.00 

0.10205 

1.08596 

0.48581 

-0.67513 

0.40000 

0.47291 

70.00 

0.15241 

1.15441 

0.54758 

-0.56115 

1.60000 

0.79410 

75.00 

0.19833 

1.22527 

0.60934 

-0.44167 

1.60000 

0.79410 

80.00 

0.26398 

1.33124 

0.68627 

-0.29822 

1.60000 

0.79410 

85.00 

0.32332 

1.44223 

0.79643 

-0.11645 

1.60000 

0.79410 

90.00 

0.39659 

1.55806 

0.90904 

0.16985 

3.60000 

0.94221 

95.00 

0.51619 

1.82081 

1.07517 

0.54825 

3.60000 

0.94221 

99.00 

0.69592 

2.25392 

1.46316 

1.36536 

6.40000 

0.98862 

100.00 

0.91614 

3.22417 

1.84403 

2.39093 

10.00000 

0.99840 
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Table  B.4:  Statistics  for  6x6  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

Mean 

0.01263 

0.99995 

0.41590 

-0.49314 

0.94327 

Var. 

0.06343 

0.14579 

0.10820 

0.60896 

1.60581 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-0.80077 

0.17810 

0.00009 

-1.67330 

0.06667 

1.00 

-0.59972 

0.33211 

0.00395 

-1.58005 

0.06667 

5.00 

-0.39328 

0.46555 

0.03713 

-1.38502 

0.06667 

10.00 

-0.31143 

0.54101 

0.07169 

-1.27677 

0.06667 

15.00 

-0.24637 

0.61323 

0.09901 

-1.18561 

0.06667 

20.00 

-0.20196 

0.66488 

0.12727 

-1.13122 

0.06667 

25.00 

-0.15682 

0.72344 

0.15999 

-1.04351 

0.06667 

30.00 

-0.12363 

0.76712 

0.19933 

-0.98299 

0.06667 

35.00 

-0.09277 

0.80950 

0.23019 

-0.91018 

0.06667 

40.00 

-0.05996 

0.86113 

0.26133 

-0.83825 

0.06667 

45.00 

-0.02497 

0.90366 

0.29669 

-0.74237 

0.60000 

50.00 

0.01349 

0.95213 

0.34466 

-0.66334 

0.60000 

55.00 

0.04299 

1.00447 

0.37927 

-0.57844 

0.60000 

60.00 

0.07803 

1.04919 

0.41911 

-0.48622 

0.60000 

65.00 

0.10933 

1.10115 

0.48232 

-0.37550 

0.60000 

70.00 

0.14049 

1.16702 

0.52592 

-0.25475 

0.60000 

75.00 

0.17750 

1.21933 

0.59073 

-0.16493 

1.66667 

80.00 

0.22375 

1.30232 

0.66756 

-0.01498 

1.66667 

85.00 

0.27933 

1.38968 

0.75484 

0.21719 

1.66667 

90.00 

0.35362 

1.51700 

0.86755 

0.54332 

3.26667 

95.00 

0.43224 

1.72053 

1.04184 

1.02088 

3.26667 

99.00 

0.57477 

2.05082 

1.46593 

2.18778 

5.40000 

00.00 

0.81547 

2.59706 

1.99716 

3.93414 

8.06667 

PC2C 

0.50159 

0.07561 


PC2C 

0.20375 

0.20375 

0.20375 

0.20375 

0.20375 

0.20375 

0.20375 

0.20375 

0.20375 

0.20375 

0.56142 

0.56142 

0.56142 

0.56142 

0.56142 

0.56142 

0.80330 

0.80330 

0.80330 

0.92931 

0.92931 

0.97984 

0.99548 


Table  B.5:  Statistics  for  7x7  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

Mean 

-0.00685 

0.99978 

0.35359 

-0.40782 

1.04398 

Var. 

0.04849 

0.10713 

0.07851 

0.50571 

2.22114 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

0.09 

-0.67332 

0.29053 

0.00008 

-1.71364 

0.04762 

1.00 

-0.53068 

0.35914 

0.00354 

-1.41901 

0.04762 

5.00 

-0.37841 

0.53304 

0.02898 

-1.23956 

0.04762 

10.00 

-0.29187 

0.60891 

0.05234 

-1.12762 

0.04762 

15.00 

-0.24113 

0.67137 

0.08520 

-1.02577 

0.04762 

20.00 

-0.19271 

0.72862 

0.11624 

-0.94829 

0.04762 

25.00 

-0.15344 

0.76641 

0.14051 

-0.88160 

0.04762 

30.00 

-0.11447 

0.80023 

0.17341 

-0.82695 

0.04762 

35.00 

-0.08695 

0.83666 

0.19888 

-0.75960 

0.42857 

40.00 

-0.06451 

0.87623 

0.22866 

-0.69334 

0.42857 

45.00 

-0.03467 

0.91916 

0.26278 

-0.61617 

0.42857 

50.00 

-0.00662 

0.96781 

0.29473 

-0.55072 

0.42857 

55.00 

0.01703 

1.00652 

0.32640 

-0.47841 

0.42857 

60.00 

0.05070 

1.05781 

0.36451 

-0.40761 

0.42857 

65.00 

0.07675 

1.10792 

0.40246 

-0.32018 

1.19048 

70.00 

0.10614 

1.15263 

0.44116 

-0.21973 

1 . 19048 

75.00 

0.13831 

1.20523 

0.49512 

-0.11362 

1.19048 

80.00 

0.17992 

1.26851 

0.54772 

0.00127 

1.19048 

85.00 

0.22121 

1.32878 

0.62165 

0.21134 

2.33333 

90.00 

0.27473 

1.43204 

0.72504 

0.48597 

2.33333 

95.00 

0.35174 

1.58285 

0.89673 

0.88641 

3.85714 

99.00 

0.50401 

1.92196 

1.27804 

2.05589 

5.76190 

.00.00 

0.66945 

2.99929 

1.97804 

5.63063 

10.71429 

PC2C 

0.50935 

0.08004 


PC2C 

0.17274 

0.17274 

0.17274 

0.17274 

0.17274 

0.17274 

0.17274 

0.17274 

0.48730 

0.48730 

0.48730 

0.48730 

0.48730 

0.48730 

0.72476 

0.72476 

0.72476 

0.72476 

0.87336 

0.87336 

0.95045 

0.98363 

0.99897 
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Table  B.6:  Statistics  for  8x8  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

Mean 

-0.02209 

0.99919 

0.31221 

-0.32576 

1.06286 

Var . 

0.03935 

0.07266 

0.06260 

0.44503 

2.22502 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-0.66801 

0.37311 

0.00045 

-1.54153 

0.00000 

1.00 

-0.49556 

0.48682 

0.00289 

-1.29857 

0.00000 

5.00 

-0.34508 

0.59135 

0.02066 

-1.11343 

0.00000 

10.00 

-0.27497 

0.67435 

0.04049 

-1.01090 

0.00000 

15.00 

-0.22425 

0.72997 

0.06606 

-0.91962 

0.14286 

20.00 

-0.18603 

0.77044 

0.09118 

-0.83390 

0.14286 

_5.00 

-0.15556 

0.80423 

0.11678 

-0.79278 

0.14286 

30.00 

-0.12424 

0.83962 

0.14562 

-0.73583 

0.14286 

35.00 

-0.09716 

0.86489 

0.17733 

-0.66327 

0.14286 

40.00 

-0.07501 

0.90692 

0.19934 

-0.60069 

0.14286 

45.00 

-0.05254 

0.94025 

0.22824 

-0.55020 

0.57143 

50.00 

-0.02538 

0.96787 

0.25723 

-0.48349 

0.57143 

55.00 

0.00127 

1.00376 

0.29122 

-0.41031 

0.57143 

60.00 

0.02596 

1.03904 

0.31806 

-0.31743 

0.57143 

65.00 

0.05247 

1.08563 

0.35097 

-0.22917 

1.28571 

70.00 

0.08324 

1.12089 

0.39208 

-0.12779 

1.28571 

75.00 

0.11103 

1.16879 

0.44365 

-0.01892 

1.28571 

80.00 

0.14531 

1.22073 

0.50085 

0.12418 

1.28571 

85.00 

0.18241 

1.28877 

0,57342 

0.28232 

2.28571 

90.00 

0.23320 

1.36823 

0.65184 

0.54521 

2.28571 

95.00 

0.31002 

1.48722 

0.80659 

0.90316 

3.57143 

99.00 

0.41740 

1.69588 

1.06319 

1.96780 

7.00000 

00.00 

0.66026 

1.91652 

1.48894 

2.98311 

17.28572 

PC2C 

0.50434 

0.09222 


PC2C 

0.00000 

0.00000 

0.00000 

0.00000 

0.29455 

0.29455 

0.29455 

0.29455 

0.29455 

0.29455 

0.55031 

0.55031 

0.55031 

0.55031 

0.74315 

0.74315 

0.74315 

0.74315 

0.86943 

0.86943 

0.94121 

0.99182 

1.00000 
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Table  B.7:  Statistics  for  9x9  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

PC2C 

Mean 

0.01449 

1.01835 

0.29373 

-0.22450 

0.97656 

0.48951 

Var. 

0.02725 

0.05813 

0.05616 

0.42751 

1.77247 

0.09142 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

PC2C 

0.09 

-0.53123 

0.43106 

0.00250 

-1.49912 

0.00000 

0.00000 

1.00 

-0.35780 

0.55253 

0.00539 

-1.22275 

0.00000 

0.00000 

5.00 

-0.24785 

0.66171 

0.02279 

-1.01732 

0.00000 

0.00000 

10.00 

-0.20140 

0.72375 

0.03931 

-0.88921 

0.00000 

0.00000 

15.00 

-0.16118 

0.76853 

0.06268 

-0.80626 

0.11111 

0.26112 

20.00 

-0.13230 

0.80759 

0.08200 

-0.74280 

0.11111 

0.26112 

25.00 

-0.09876 

0.84645 

0.10157 

-0.68282 

0.11111 

0.26112 

30.00 

-0.07034 

0.87943 

0.12781 

-0.60883 

0.11111 

0.26112 

35.00 

-0.04329 

0.91529 

0.15626 

-0.53436 

0.11111 

0.26112 

40.00 

-0.02005 

0.94106 

0.17948 

-0.48167 

0.44444 

0.49501 

45.00 

-0.00274 

0.96573 

0.21440 

-0.41542 

0.44444 

0.49501 

50.00 

0.01365 

0.99587 

0.23989 

-0.34401 

0.44444 

0.49501 

55.00 

0.03329 

1.02532 

0.27495 

-0.25978 

0.44444 

0.49501 

60.00 

0.05924 

1.05682 

0.30775 

-0.18932 

0.44444 

0.49501 

65.00 

0.07931 

1.09183 

0.34275 

-0.13489 

1 . 00000 

0.68269 

70.00 

0.09891 

1.12798 

0.38146 

-0.04237 

1.00000 

0.68269 

75.00 

0.12230 

1.17419 

0.42105 

0.06178 

1.00000 

0.68269 

80.00 

0.15316 

1.21673 

0.47502 

0.20951 

1.77778 

0.81758 

85.00 

0.18539 

1.26821 

0.53255 

0.38286 

1.77778 

0.81758 

90.00 

0.22856 

1.33448 

0.61060 

0.57001 

2.77778 

0.90440 

95.00 

0.28052 

1.43599 

0.76976 

0.94686 

4.00000 

0.95448 

99.00 

0.41290 

1.63916 

1.03606 

2.10575 

5.44444 

0.98035 

.00.00 

0.62502 

2.09891 

1.34140 

3.87976 

9.00000 

0.99732 
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Table  B.8:  Statistics  for  10x10  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

Mean 

0.00262 

1.00007 

0.26071 

-0.17996 

0.95022 

Var . 

0.02100 

0.04811 

0.04242 

0.38934 

1.60737 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

0.09 

-0.43397 

0.48739 

0.00012 

-1.26288 

0.02222 

1.00 

-0.33856 

0.55785 

0.00417 

-1.15601 

0.02222 

5.00 

-0.23587 

0.68262 

0.02242 

-0.96351 

0.02222 

10.00 

-0.17603 

0.73338 

0.04044 

-0.85357 

0.02222 

15.00 

-0.14537 

0.77469 

0.06335 

-0.76318 

0.02222 

20.00 

-0.12290 

0.81386 

0.08486 

-0.67927 

0.02222 

25.00 

-0.09683 

0.85235 

0.09971 

-0.61801 

0.20000 

30.00 

-0.07910 

0.87587 

0.12149 

-0.55201 

0.20000 

35.00 

-0.06054 

0.90122 

0.14349 

-0.49073 

0.20000 

40.00 

-0.03980 

0.92963 

0.15987 

-0.43649 

0.20000 

45.00 

-0.01910 

0.95341 

0.18403 

-0.37625 

0.20000 

50.00 

-0.00197 

0.97973 

0.21120 

-0.30165 

0.55556 

55.00 

0.01947 

1.00672 

0.23793 

-0.22638 

0.55556 

60.00 

0.03844 

1.03021 

0.26355 

-0.15477 

0.55556 

65.00 

0.06128 

1.06098 

0.29910 

-0.07801 

1.08889 

70.00 

0.07895 

1.08918 

0.33435 

0.01126 

1.08889 

75.00 

0.10213 

1.13725 

0. 30746 

0.11909 

1.08889 

80.00 

0.12860 

1.18201 

0.41511 

0.24380 

1.80000 

85.00 

0.15380 

1.22437 

0.47418 

0.41395 

1.80000 

90.00 

0.18695 

1.30206 

0.56094 

0.64412 

2.68889 

95.00 

0.24350 

1.38299 

0.68496 

1.05637 

3.75556 

99.00 

0.34831 

1.58461 

0.86535 

1.78730 

6.42222 

00.00 

0.43478 

1.92315 

1.17668 

2.71917 

9.80000 

PC2C 

0.49854 

0.08058 


PC2C 

0.11850 

0.11850 

0.11850 

0.11850 

0.11850 

0.11850 

0.34528 

0.34528 

0.34528 

0.34528 

0.34528 

0.54394 

0.54394 

0.54394 

0.70328 

0.70328 

0.70328 

0.82029 

0.82029 

0.89893 

0.94736 

0.98872 

0.99823 
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Table  B.9:  Statistics  for  11x11  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

IGAMMA1I 

GAMMA2 

CHISQC 

Mean 

-0.00051 

0.99066 

0.24650 

-0.14316 

1.03081 

Var . 

0.01891 

0.03521 

0.03950 

0.34739 

2.18882 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

|GAMMA1 | 

GAMMA2 

CHISQC 

-0.51548 

0.45546 

0.00011 

-1.18467 

0.01818 

1.00 

-0.31148 

0.60322 

0.00381 

-1.04224 

0.01818 

5.00 

-0.22131 

0.71343 

0.01598 

-0.85125 

0.01818 

10.00 

-0.17210 

0.76451 

0.03563 

-0.74136 

0.01818 

15.00 

-0.14810 

0.79693 

0.05786 

-0.66022 

0.01818 

20.00 

-0.12215 

0.82530 

0.07945 

-0.59152 

0.01818 

25.00 

-0.09846 

0.85212 

0.10065 

-0.53103 

0.16364 

30.00 

-0.07260 

0.88324 

0.11765 

-0.47671 

0.16364 

35.00 

-0.05552 

0.91041 

0.13455 

-0.42756 

0.16364 

40.00 

-0.03606 

0.93018 

0.15819 

-0.36971 

0.16364 

45.00 

-0.01960 

0.95275 

0.17511 

-0.31768 

0.45455 

50.00 

-0.00339 

0.97607 

0.2005! 

-0.27210 

0.45455 

55.00 

0.01445 

1.00038 

0.22449 

-0.20735 

0.45455 

60.00 

0.03177 

1.02542 

0.25589 

-0.14704 

0.89091 

65.00 

0.04779 

1.05562 

0.28618 

-0.07422 

0.89091 

70.00 

0.06635 

1.08538 

0.31650 

0.02158 

1.47273 

75.00 

0.09450 

1.11543 

0.34498 

0.11200 

1.47273 

80.00 

0.12079 

1.14351 

0.37950 

0.24114 

1.47273 

85.00 

0.14799 

1.18101 

0.43569 

0.36459 

2 . 20000 

90.00 

0.17937 

1.23455 

0.50733 

0.62537 

2 . 20000 

95.00 

0.22935 

1.31975 

0.61529 

0.94102 

4.09091 

99.00 

0.30881 

1.46826 

0.88918 

1.88365 

6.56364 

00.00 

0.45266 

1.84344 

1.35711 

4.09483 

15.29091 

PC2C 

0.50232 

0.08496 


PC2C 

0.10726 

0.10726 

0.10726 

0.10726 

0.10726 

0.10726 

0.31417 

0.31417 

0.31417 

0.31417 

0.49981 

0.49981 

0.49981 

0.65478 

0.65478 

0.77507 

0.77507 

0.77507 

0.86199 

0.86199 

0.95686 

0.98958 

1.00000 
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Table  B.10:  Statistics  for  12x12  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

Mean 

0.00317 

0.99844 

0.22751 

-0.14736 

0.93565 

Var. 

0.01498 

0.02939 

0.03075 

0.25186 

1.56145 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-0.40626 

0.48667 

0.00007 

-1.23879 

0.00000 

1.00 

-0.28197 

0.63963 

0.00225 

-1.03516 

0.00000 

5.00 

-0.19525 

0.73649 

0.01976 

-0.78969 

0.00000 

10.00 

-0.15639 

0.78705 

0.03619 

-0.67503 

0.00000 

15.00 

-0.12901 

0.82425 

0.05772 

-0.61168 

0.06061 

20.00 

-0.10645 

0.85001 

0.07459 

-0.54909 

0.06061 

25.00 

-0.08183 

0.87798 

0.08856 

-0.49824 

0.06061 

30.00 

-0.06067 

0.90359 

0.10866 

-0.44408 

0.06061 

35.00 

-0.04601 

0.92192 

0.12839 

-0.38689 

0.24242 

40.00 

-0.02698 

0.94608 

0.14722 

-0.33457 

0.24242 

45.00 

-0.01305 

0.96607 

0.16980 

-0.27925 

0.24242 

50.00 

0.00426 

0.98645 

0.18871 

-0.22100 

0.54545 

55.00 

0.02164 

1.00884 

0.21072 

-0.17254 

0.54545 

60.00 

0.03465 

1.02919 

0.23718 

-0.11231 

0.54545 

65.00 

0.04837 

1.05591 

0.26092 

-0.06199 

0.96970 

70.00 

0.06840 

1.08015 

0.28974 

0.02035 

0.96970 

75.00 

0.08770 

1.10690 

0.32401 

0.09299 

1.51515 

80.00 

0.10576 

1.13416 

0.36783 

0.19168 

1.51515 

85.00 

0.12424 

1.16804 

0.41086 

0.31436 

2.18182 

90.00 

0.16033 

1.22026 

0.47491 

0.50835 

2.18182 

95.00 

0.21158 

1.31458 

0.55566 

0.78019 

3.87879 

99.00 

0.29036 

1.43828 

0.74381 

1.45655 

6.06061 

00.00 

0.34516 

1.65202 

1.12910 

2.49610 

8.72727 

PC2C 

0.48747 

0.08772 


PC2C 

0.00000 

0.00000 

0.00000 

0.00000 

0.19446 

0.19446 

0.19446 

0.19446 

0.37754 

0.37754 

0.37754 

0.53982 

0.53982 

0.53982 

0.67524 

0.67524 

0.78163 

0.78163 

0.86035 

0.86035 

0.95109 

0.98619 

0.99688 
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Table  B.ll:  Statistics  for  13x13  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

Mean 

-0.00289 

1.00742 

0.20037 

-0.11891 

1.01855 

Var . 

0.01359 

0.02814 

0.02408 

0.22939 

1.82731 

Percentage  Points  of 

Statistics 

% 

XBAR 

VAR 

|GAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-0.45176 

0.57806 

0.00089 

-1.07879 

0.00000 

1.00 

-0.27757 

0.65076 

0.00477 

-0.91152 

0.00000 

5.00 

-0.19005 

0.74951 

0.01741 

-0.74117 

0.00000 

10.00 

-0.15093 

0.79656 

0.03183 

-0.64397 

0.05128 

15.00 

-0.12572 

0.83284 

0.04610 

-0.57226 

0.05128 

20.00 

-0.10341 

0.86822 

0.06395 

-0.51109 

0.05128 

25.00 

-0.07915 

0.89583 

0.08160 

-0.46301 

0.05128 

30.00 

-0.06337 

0.91651 

0.09445 

-0.40471 

0.20513 

35.00 

-0.04842 

0.93694 

0.11177 

-0.35677 

0.20513 

40.00 

-0.03446 

0.95946 

0.13141 

-0.31242 

0.20513 

45.00 

-0.01880 

0.97768 

0.14856 

-0.24783 

0.46154 

50.00 

-0.00316 

0.99721 

0.16699 

-0.20334 

0.46154 

55.00 

0.01052 

1.01685 

0.18977 

-0.14181 

0.46154 

60.00 

0.02518 

1.03787 

0.20939 

-0.08601 

0.82051 

65.00 

0.03965 

1.06120 

0.22979 

-0.01864 

0.82051 

70.00 

0.05656 

1.08623 

0.25475 

0.05118 

1.28205 

75.00 

0.07491 

1.10955 

0.28372 

0.12370 

1.28205 

80.00 

0.09576 

1.14168 

0.31748 

0.21308 

1.84615 

85.00 

0.11633 

1.17610 

0.35289 

0.34022 

1.84615 

90.00 

0.15044 

1.22677 

0.41635 

0.51854 

2.51282 

95.00 

0.18829 

1.30374 

0.48979 

0.80724 

3.28205 

99.00 

0.28062 

1.45031 

n. 69819 

1.45445 

6.20513 

00.00 

0.36349 

1.65806 

0.92051 

2.31334 

10.0512P 

PC2C 

0.50906 

0.08472 


PC2C 

0.00000 

0.00000 

0.00000 

0.17915 

0.17915 

0.17915 

0.17915 

0.34939 

0.34939 

0.34939 

0.50309 

0.50309 

0.50309 

0.63498 

0.63498 

0.74248 

0.74248 

0.82578 

0.82578 

0.88707 

0.92997 

0.98729 

0.99844 
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Table  B.12:  Statistics  for  14x14  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

Mean 

-0.00031 

0.99516 

0.19272 

-0.08523 

1.00072 

Var. 

0.01023 

0.02157 

0.02311 

0.22773 

1.99211 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

0.09 

-0.35076 

0.60193 

0.00003 

-1.05385 

0.01099 

1.00 

-0.22857 

0.68912 

0.00229 

-0.87129 

0.01099 

5.00 

-0.16285 

0.76230 

0.01510 

-0.71432 

0.01099 

10.00 

-0.12604 

0.81425 

0.02792 

-0.61038 

0.01099 

15.00 

-0.10498 

0.84496 

0.04603 

-0.53496 

0.01099 

20.00 

-0.08358 

0.86639 

0.05997 

-0.48629 

0.09890 

25.00 

-0.06825 

0.89039 

0.07548 

-0.42127 

0.09890 

30.00 

-0.05514 

0.91382 

0.09293 

-0.36969 

0.27473 

35.00 

-0.04209 

0.93261 

0.10944 

-0.32557 

0.27473 

40.00 

-0.03116 

0.95055 

0.12512 

-0.28071 

0.27473 

45.00 

-0.01568 

0.96782 

0.14141 

-0.22614 

0.27473 

50.00 

-0.00232 

0.99036 

0.15719 

-0.17459 

0.53846 

55.00 

0.01055 

1.00706 

0.17564 

-0.10879 

0.53846 

60.00 

0.02484 

1.02729 

0.19483 

-0.05332 

0.89011 

65.00 

0.03730 

1.04497 

0.22151 

0.01733 

0.89011 

70.00 

0.05250 

1.06213 

0.24742 

0.08995 

0.89011 

75.00 

0.06842 

1.08871 

0.27657 

0.17671 

1.32967 

80.00 

0.08527 

1.11075 

0.30448 

0.27013 

1.32967 

85.00 

0.10561 

1.14197 

0.34690 

0.37705 

1.85714 

90.00 

0.12708 

1.18061 

0.39562 

0.57470 

2.47253 

95.00 

0.16815 

1.24696 

0.48065 

0.82258 

3.96703 

99.00 

0.23796 

1.37935 

0.67250 

1.31528 

6.86813 

00.00 

0.31674 

1.66742 

1.03333 

1.98715 

13.46154 
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PC2C 

0.53039 

0.07739 


PC2C 

0.08349 

0.08349 

0.08349 

0.08349 

0.08349 

0.24685 

0.24685 

0.39982 

0.39982 

0.39982 

0.39982 

0.53693 

0.53693 

0.65456 

0.65456 

0.65456 

0.75113 

0.75113 

0.82705 

0.88414 

0.95358 

0.99120 

1.00000 


Table  B.13:  Statistics  for  15x15  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

Mean 

-0.00505 

0.99898 

0.17836 

-0.09901 

0.99626 

Var . 

0.00952 

0.01852 

0.01825 

0.18072 

1.90045 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

IGAMMA1 | 

GAMMA2 

CHISQC 

0.09 

-0.34322 

0.59349 

0.00031 

-0.96264 

0.00952 

1.00 

-0.23798 

0.70732 

0.00176 

-0.83563 

0.00952 

5.00 

-0.16632 

0.78187 

0.01192 

-0.65822 

0.00952 

10.00 

-0.12957 

0.82348 

0.02790 

-0.55566 

0.00952 

15.00 

-0.10309 

0.85008 

0.04167 

-0.48942 

0.00952 

20.00 

-0.08475 

0.87971 

0.05541 

-0.44536 

0.08571 

25.00 

-0.06952 

0.90345 

0.07284 

-0.39342 

0.08571 

30.00 

-0.05583 

0.92299 

0.08492 

-0.34364 

0.08571 

35.00 

-0.03950 

0.94318 

0.10697 

-0.29604 

0.23810 

40.00 

-0.02736 

0.96182 

0.12441 

-0.24931 

0.23810 

45.00 

-0.01782 

0.98107 

0.13925 

-0.20931 

0.46667 

50.00 

-0.00856 

0.99639 

0.15352 

-0.16881 

0.46667 

55.00 

0.00533 

1.01569 

0.16746 

-0.12534 

0.46667 

60.00 

0.01799 

1.03366 

0.18288 

-0.07379 

0.77143 

65.00 

0.03149 

1.04870 

0.20425 

-0.01126 

0.77143 

70.00 

0.04556 

1.06998 

0.22796 

0.05157 

1.15238 

75.00 

0.05867 

1.09146 

0.25112 

0.11730 

1.15238 

80.00 

0.07748 

1.11635 

0.28039 

0.18541 

1.60952 

85.00 

0.09769 

1.13859 

0.31997 

0.28203 

2.14286 

90.00 

0.12073 

1.17763 

0.36422 

0.42582 

2.75238 

95.00 

0.15496 

1.21867 

0.44414 

0.71132 

4.20000 

99.00 

0.22253 

1.32212 

0.58923 

1.28618 

5.95238 

00.00 

0.36057 

1.49480 

0.83878 

2.30351 

11.66667 

PC2C 

0.50433 

0.08177 


PC2C 

0.07774 

0.07774 

0.07774 

0.07774 

0.07774 

0.23030 

0.23030 

0.23030 

0.37441 

0.37441 

0.50547 

0.50547 

0.50547 

0.62023 

0.62023 

0.71695 

0.71695 

0.79544 

0.85679 

0.90287 

0.95955 

0.98532 

0.99938 
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Table  B.14:  Statistics  for  16x16  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

Mean 

0.00583 

1.00341 

0.16866 

-0.06000 

1.00767 

Var. 

0.00796 

0.01732 

0.01766 

0.17154 

1.79328 

Percentage  Points  of 

Statistics 

% 

XBAR 

VAR 

1 GAMMA1 | 

GAMMA2 

CHISQC 

-0.26300 

0.67478 

0.00001 

-0.86597 

0.00000 

1.00 

-0.20974 

0.73698 

0.00231 

-0.79116 

0.00000 

5.00 

-0.14451 

0.79708 

0.01362 

-0.61372 

0.00000 

10.00 

-0.11249 

0.83376 

0.02664 

-0.53458 

0.03333 

15.00 

-0.08577 

0.86932 

0.03738 

-0.47459 

0.03333 

20.00 

-0.06926 

0.89058 

0.04815 

-0.40758 

0.13333 

25.00 

-0.05515 

0.91187 

0.06274 

-0.35417 

0.13333 

30.00 

-0.03893 

0.93082 

0.07768 

-0.30641 

0.13333 

35.00 

-0.02582 

0.94556 

0.09239 

-0.25421 

0.30000 

40.00 

-0.01390 

0.96438 

0.10944 

-0.21684 

0.30000 

45.00 

-0.00520 

0.97880 

0.12428 

-0.16532 

0.53333 

50.00 

0.00557 

0.99551 

0.14173 

-0.11349 

0.53333 

55.00 

0.01554 

1.01314 

0.15644 

-0.06586 

0.53333 

60.00 

0.02615 

1.03212 

0.17244 

-0.01288 

0.83333 

65.00 

0.04032 

1.05063 

0.19234 

0.03655 

0.83333 

70.00 

0.05166 

1.06566 

0.21618 

0.09285 

1.20000 

75.00 

0.06620 

1.09146 

0.24053 

0.15898 

1.20000 

80.00 

0.07844 

1.11650 

0.27022 

0.25764 

1.63333 

85.00 

0.09739 

1.13907 

0.30396 

0.34723 

2.13333 

90.00 

0.12409 

1.16847 

0.35211 

0.47592 

2.70000 

95.00 

0.15635 

1.22838 

0.44211 

0.71106 

4.03333 

99.00 

0.19254 

1.34582 

0.54155 

1.19377 

6.53333 

00.00 

0.29737 

1.46845 

0.76109 

1.62261 

10.80000 

PC2C 

0.51146 

0.08206 


PC2C 

0.00000 

0.00000 

0.00000 

0.14487 

0.14487 

0.28500 

0.28500 

0.28500 

0.41612 

0.41612 

0.53479 

0.53479 

0.53479 

0.63870 

0.63870 

0.72667 

0.72667 

0.79876 

0.85589 

0.89963 

0.95537 

0.98940 

0.99902 
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Table  B.15:  Statistics  for  17x17  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

Mean 
Var . 

-0.00256 

0.00760 

1.00037 

0.01557 

0.15783 

0.01447 

-0.07865 

0.13073 

0.99650 

2.09215 

Percentage 

Points  of 

Statistics 

Z 

XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

0.09 

-0.29859 

0.61413 

0.00001 

-0.94676 

0.00000 

1.00 

-0.21350 

0.74408 

0.00295 

-0.74785 

0.00000 

5.00 

-0.14132 

0.79674 

0.01368 

-0.57328 

0.00000 

10.00 

-0.11517 

0.84231 

0.02411 

-0.49074 

0.02941 

15.00 

-0.09192 

0.86862 

0.03715 

-0.43501 

0.02941 

20.00 

-0.07324 

0.89794 

0.05055 

-0.36967 

0.02941 

25.00 

-0.05954 

0.91764 

0.06312 

-0.32743 

0.11765 

30.00 

-0.04888 

0.93505 

0.07653 

-0.28258 

0.11765 

35.00 

-0.03730 

0.95022 

0.08804 

-0.24561 

0.26471 

40.00 

-0.02741 

0.96383 

0.10574 

-0.20445 

0.26471 

45.00 

-0.01602 

0.97692 

0.12380 

-0.15966 

0.47059 

50.00 

-0.00331 

0.99388 

0.13666 

-0.12009 

0.47059 

55.00 

0.00832 

1.00997 

0.14891 

-0.08359 

0.47059 

60.00 

0.01935 

1.02167 

0.16520 

-0.04090 

0.73529 

65.00 

0.02974 

1.03784 

0.18127 

0.00162 

0.73529 

70.00 

0.04111 

1.05759 

0.20172 

0.05722 

1.05882 

75.00 

0.05413 

1.07908 

0.22479 

0.11378 

1.44118 

80.00 

0.06885 

1.10203 

0.25276 

0.19037 

1.88235 

85.00 

0.08916 

1.13243 

0.28010 

0.27082 

1.88235 

90.00 

0.10681 

1.15957 

0.31593 

0.39650 

2.38235 

95.00 

0.13616 

1.21412 

0.38469 

0.57191 

3.55882 

99.00 

0.20998 

1.31564 

0.49904 

1.02926 

7.52941 

.00.00 

0.29930 

1.44939 

0.81570 

1.99064 

14.23529 

PC2C 

0.49998 

0.08371 


PC2C 

0.00000 

0.00000 

0.00000 

0.13617 

0.13617 

0.13617 

0.26840 

0.26840 

0.39309 

0.39309 

0.50728 

0.50728 

0.50728 

0.60883 

0.60883 

0.69651 

0.77004 

0.82994 

0.82994 

0.87728 

0.94076 

0.99395 

1.00000 
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Table  B.16:  Statistics  for  18x18  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

Mean 

0.00119 

0.99752 

0.15011 

-0.05410 

0.  '*6302 

Var . 

0.00619 

0.01273 

0.01441 

0.13448 

1 . >3580 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

0.09 

-0.24342 

0.64337 

0.00003 

-0.87953 

0.00654 

1.00 

-0.18399 

0.75072 

0.00173 

-0.69734 

0.00654 

5.00 

-0.13070 

0.81457 

0.01081 

-0.57235 

0.00654 

10.00 

-0.10238 

0.85104 

0.02228 

-0.48790 

0.00654 

15.00 

-0.08219 

0.88080 

0.03350 

-0.40802 

0.05882 

20.00 

-0.06394 

0.89903 

0.04353 

-0.35682 

0.05882 

25.00 

-0.05095 

0.91891 

0.05776 

-0.31693 

0.05882 

30.00 

-0.03953 

0.93641 

0.06877 

-0.27139 

0.16340 

35.00 

-0.02876 

0.94880 

0.08181 

-0.22359 

0.16340 

40.00 

-0.01984 

0.96639 

0.09492 

-0.18152 

0.32026 

45.00 

-0.00759 

0.98290 

0.10902 

-0.14667 

0.32026 

50.00 

0.00274 

0.99540 

0.12123 

-0.10436 

0.52941 

55.00 

0.01213 

1.00842 

0.13685 

-0.06505 

0.52941 

60.00 

0.02348 

1.02387 

0.15036 

-0.02419 

0.79085 

65.00 

0.03379 

1.03866 

0.16932 

0.03692 

0.79085 

70.00 

0.04459 

1.05609 

0.18966 

0.08985 

1 . 10458 

75.00 

0.05607 

1.07408 

0.21752 

0.14895 

1.47059 

80.00 

0.06812 

1.09432 

0.24109 

0.22955 

1.47059 

85.00 

0.08225 

1.11375 

0.27762 

0.30823 

1.88889 

90.00 

0.10022 

1.14764 

0.31699 

0.42514 

2.35948 

95.00 

0.12205 

1.18431 

0.37345 

0.60672 

3.45752 

99.00 

0.17851 

1.25472 

0.51895 

1.11051 

6.28105 

00.00 

0.28168 

1.44906 

0.66495 

1.48588 

8.94771 

PC  2C 
0.49530 
0.08330 


PC2C 

0.06443 

0.06443 

0.06443 

0.06443 

0.19164 

0.19164 

0.19164 

0.31395 

0.31395 

0.42855 

0.42855 

0.53315 

0.53315 

0.62616 

0.62616 

0.70673 

0.77473 

0.77473 

0.83068 

0.87547 

0.93706 

0.98782 

0.99724 
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Table  B.17:  Statistics  for  19x19  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

|GAMMA1 | 

GAMMA2 

CHISQC 

Mean 

0.00131 

1.00218 

0.14682 

-0.04358 

1.00357 

Var. 

0.00586 

0.01185 

0.01288 

0.12249 

1.78927 

Percentage  Points  of  Statistics 


% 

XBAR 

VAR 

0.09 

-0.24941 

0.72855 

1.00 

-0.16690 

0.77533 

5.00 

-0.12470 

0.82701 

10.00 

-0.09730 

0.86152 

15.00 

-0.07941 

0.88981 

20.00 

-0.06613 

0.90993 

25.00 

-0.05364 

0.92526 

30.00 

-0.04201 

0.94168 

35.00 

-0.03034 

0.95600 

40.00 

-0.01697 

0.97091 

45.00 

-0.00489 

0.98491 

50.00 

0.00347 

0.99695 

55.00 

0.01273 

1.00920 

60.00 

0.02217 

1.02502 

65.00 

0.03196 

1.04027 

70.00 

0.04192 

1.05732 

75.00 

0.05422 

1.07276 

80.00 

0.06517 

1.09282 

85.00 

0.07757 

1.11318 

90.00 

0.09479 

1.14339 

95.00 

0.12552 

1.19870 

99.00 

0.18480 

1.25209 

100.00 

0.26682 

1.45466 

(GAMMA1 ( 

GAMMA2 

CHISQC 

0.00008 

-0.81430 

0.00585 

0.00165 

-0.66645 

0.00585 

0.01233 

-0.53454 

0.00585 

0.02389 

-0.45290 

0.00585 

0.03503 

-0.38347 

0.05263 

0.04588 

-0.33176 

0.05263 

0.05482 

-0.29317 

0.14620 

0.06893 

-0.25376 

0.14620 

0.08098 

-0.20123 

0.14620 

0.09404 

-0.16449 

0.28655 

0.10755 

-0.12827 

0.28655 

0.12133 

-0.08610 

0.47368 

0.13792 

-0.04254 

0.70760 

0.15157 

-0.00261 

0.70760 

0.16889 

0.04254 

0.98830 

0.18660 

0.09737 

0.98830 

0.21204 

0.15995 

1.31579 

0.23071 

0.21138 

1 . 69006 

0.26844 

0.28734 

2.11111 

0.31156 

0.41861 

3.09357 

0.36531 

0.61609 

3.65497 

0.49018 

1.01420 

5.61988 

0.57527 

1.57119 

11.84210 

PC2C 

0.50582 

0.08366 


PC2C 

0.06096 

0.06096 

0.06096 

0.06096 

0.18145 

0.18145 

0.29781 

0.29781 

0.29781 

0.40756 

0.40756 

0.50870 

0.59976 

0.59976 

0.67984 

0.67984 

0.74864 

0.80641 

0.85379 

0.92141 

0.94409 

0.98221 

0.99944 
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Table  B.18:  Statistics  for  20x20  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

PC2C 

Mean 

-0.00062 

1.00201 

0.13699 

-0.03912 

0.94329 

0.49941 

Var. 

0.00514 

0.01061 

0.01112 

0.11477 

1.62271 

0.08101 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

PC2C 

0.09 

-0.20625 

0.66775 

0.00009 

-0.84644 

0.00000 

0.00000 

1.00 

-0.16349 

0.75965 

0.00252 

-0.64227 

0.00000 

0.00000 

5.00 

-0.11345 

0.83845 

0.01273 

-0.48679 

0.00000 

0.00000 

10.00 

-0.09148 

0.87312 

0.02216 

-0.42335 

0.02105 

0.11536 

15.00 

-0.07469 

0.89385 

0.03435 

-0.36603 

0.02105 

0.11536 

20.00 

-0.06150 

0.91058 

0.04447 

-0.31870 

0.08421 

0.22833 

25.00 

-0.05197 

0.93173 

0.05502 

-0.28177 

0.08421 

0.22833 

30.00 

-0.04122 

0.94665 

0.06556 

-0.23833 

0.18947 

0.33664 

35.00 

-0.03068 

0.95995 

0.07643 

-0.20847 

0.18947 

0.33664 

40.00 

-0.02247 

0.97115 

0.08945 

-0.17093 

0.33684 

0.43834 

45.00 

-0.01410 

0.98571 

0.10469 

-0.12803 

0.33684 

0.43834 

50.00 

-0.00362 

1.00198 

0.11492 

-0.07984 

0.52632 

0.53184 

55.00 

0.00753 

1.01492 

0.12759 

-0.04382 

0.52632 

0.53184 

60.00 

0.01814 

1.02765 

0.14248 

0.00647 

0.75789 

0.61602 

65.00 

0.02780 

1.03974 

0.15503 

0.04738 

0.75789 

0.61602 

70.00 

0.03902 

1.05427 

0.17836 

0.09982 

1.03158 

0.69021 

75.00 

0.04909 

1.07021 

0.19412 

0.14831 

1.34737 

0.75425 

80.00 

0.05930 

1.08649 

0.21651 

0.20763 

1.70526 

0.80840 

85.00 

0.07058 

1.10668 

0.24283 

0.26996 

2.10526 

0.85323 

90.00 

0.09323 

1.13387 

0.28126 

0.37784 

2.54737 

0.88951 

95.00 

0.11678 

1.17167 

0.33344 

0.59012 

3.55789 

0.94073 

99.00 

0.16961 

1.24566 

0.47241 

1.07502 

6.08421 

0.98638 

00.00 

0.27720 

1.35356 

0.67606 

1.37094 

11.13684 

0.99914 
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Table  B.19:  Statistics  for  21x21  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA1 | 

GAMMA2 

CHISQC 

Mean 

-0.00083 

0.99987 

0.12595 

-0.05188 

0.98462 

Var . 

0.00468 

0.00982 

0.00914 

0.08817 

1.82501 

Percentage 

Points  of 

Statistics 

% 

XBAR 

VAR 

| GAMMA 1| 

GAMMA2 

CHISQC 

0.09 

-0.23580 

0.66768 

0.00037 

-0.69795 

0.00000 

1.00 

-0.16453 

0.78771 

0.00193 

-0.59643 

0.00000 

5.00 

-0.11434 

0.83865 

0.00920 

-0.47746 

0.00000 

10.00 

-0.08949 

0.87272 

0.01968 

-0.40411 

0.01905 

15.00 

-0.07214 

0.89725 

0.03092 

-0.34781 

0.07619 

20.00 

-0.05975 

0.91504 

0.04101 

-0.30439 

0.07619 

25.00 

-0.04742 

0.93374 

0.05237 

-0.26478 

0.07619 

30.00 

-0.03613 

0.94968 

0.06267 

-0.22815 

0.17143 

35.00 

-0.02569 

0.96174 

0.07389 

-0.19062 

0.17143 

40.00 

-0.01707 

0.97443 

0.08600 

-0.15825 

0.30476 

45.00 

-0.00812 

0.98671 

0.09513 

-0.12436 

0.30476 

50.00 

-0.00018 

0.99819 

0.10588 

-0.08571 

0.47619 

55.00 

0.00851 

1.00818 

0.11854 

-0.04733 

0.68571 

60.00 

0.01789 

1.02120 

0.12932 

-0.00988 

0.68571 

65.00 

0.02717 

1.03577 

0.14253 

0.03381 

0.93333 

70.00 

0.03531 

1.04764 

0.15980 

0.07097 

1.21905 

75.00 

0.04606 

1.06206 

0.17700 

0.12437 

1.21905 

80.00 

0.05655 

1.08194 

0.19987 

0.18685 

1.54286 

85.00 

0.06863 

1.10267 

0.22235 

0.25149 

1.90476 

90.00 

0.08248 

1.13039 

0.26944 

0.33172 

2.74286 

95.00 

0.10986 

1.17004 

0.30720 

0.47384 

3.73333 

99.00 

0.16025 

1.24209 

0.40792 

0.73353 

6.17143 

.00.00 

0.22484 

1.34792 

0.62186 

1.44753 

11.90476 

PC2C 

0.50338 

0.08182 


PC2C 

0.00000 

0.00000 

0.00000 

0.10977 

0.21747 

0.21747 

0.21747 

0.32115 

0.32115 

0.41909 

0.41909 

0.50984 

0.59238 

0.59238 

0.66601 

0.73045 

0.73045 

0.78581 

0.83246 

0.90229 

0.94665 

0.98704 

0.99946 


B-20 


Table  B.20:  Statistics  for  22x22  Subjective  Judgment  Matrices 


Means  and  Variances  of  Statistics 


XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

Mean 

-0.00073 

0.99937 

0.12144 

-0.03658 

0.96357 

Var . 

0.00416 

0.00866 

0.00887 

0.09109 

1.66709 

Percentage 

Points  of 

Statistics 

X 

XBAR 

VAR 

| GAMMA 1 | 

GAMMA2 

CHISQC 

0.09 

-0.17272 

0.74084 

0.00022 

-0.72286 

0.00433 

1.00 

-0.15452 

0.79226 

0.00169 

-0.57188 

0.00433 

5.00 

-0.10214 

0.85113 

0.01039 

-0.44804 

0.00433 

10.00 

-0.08020 

0.88636 

0.02016 

-0.38200 

0.00433 

15.00 

-0.06620 

0.90350 

0.02796 

-0.33352 

0.03896 

20.00 

-0.05649 

0.91820 

0.03919 

-0.29574 

0.03896 

25.00 

-0.04713 

0.93336 

0.04791 

-0.25078 

0.10823 

30.00 

-0.03826 

0.94669 

0.06028 

-0.21528 

0.10823 

35.00 

-0.02640 

0.96225 

0.07054 

-0.18057 

0.21212 

40.00 

-0.01951 

0.97190 

0.08004 

-0.14391 

0.21212 

45.00 

-0.01088 

0.98338 

0.08915 

-0.10994 

0.35065 

50.00 

-0.00260 

0.99393 

0.10025 

-0.07087 

0.52381 

55.00 

0.00772 

1.00767 

0.11210 

-0.03497 

0.52381 

60.00 

0.01434 

1.02009 

0.12539 

-0.00171 

0.73160 

65.00 

0.02318 

1.03330 

0.13997 

0.04391 

0.97403 

70.00 

0.03413 

1.04460 

0.15554 

0.08195 

0.97403 

75.00 

0.04356 

1.06101 

0.17064 

0.13288 

1.25108 

80.00 

0.05393 

1.07621 

0.18966 

0.18281 

1.56277 

85.00 

0.06770 

1.09784 

0.21393 

0.25614 

1.90909 

90.00 

0.08228 

1.12374 

0.25471 

0.36081 

2.70563 

95.00 

0.10595 

1.15619 

0.30077 

0.50454 

3.64069 

99.00 

0.14983 

1.21267 

0.39351 

0.88435 

5.92641 

00.00 

0.22273 

1.32238 

0.64989 

1.19565 

11.25974 

A 


PC2C 

0.49885 

0.08270 


PC2C 

0.05246 

0.05246 

0.05246 

0.05246 

0.15647 

0.15647 

0.25783 

0.25783 

0.35489 

0.35489 

0.44625 

0.53078 

0.53078 

0.60764 

0.67632 

0.67632 

0.73665 

0.78874 

0.83295 

0.89999 

0.94361 

0.98510 

0.99919 


B.3  The  statistics  in  Tables  B.l  through  B.20  were  generated  using  Monte 
Carlo  simulation.  Each  table  is  based  on  1100  randomly-generated  matrices. 
Errors  in  estimating  entries  in  SJMs  were  assumed  to  be  ratio  errors  that  are 
log-normally  distributed,  i.e., 


a. .  =  (w./w. )e. . 
ij  i  J  il 


where  a. . 

ij 


and  e.  . 
ij 


=  (i,j)  entry  in  subjective  judgment  matrix  A, 
*  weight  for  entity  i, 

=  error  in  estimating  w^/w^. 


The  logarithmic  least-squares  approach  to  estimating  the  weights  minimizes  the 
sum  of  squares  of  the  logarithms  of  errors.  Accordingly,  the  entries  in  the 
randomly-generated  matrices  had  a  normal  distribution  with  zero  mean  and  unit 
variance . 


Statistics  Addressing  Normality  of  Errors 

B . 4  The  first  four  statistics  in  each  table,  XBAR,  VAR,  |GAMMA1|,  and  GAMMA2, 
can  be  used  to  test  whether  the  error  terms  are  normally  distributed.  Because 
we  want  to  assess  consistency  of  SJMs  by  using  an  F  test  for  the  equality  of 
variances  and  the  F  test  is  most  sensitive  to  the  skewness  of  the  error  dis¬ 
tribution  [4],  a  test  of  skewness  is  most  appropriate  for  assessing  whether 
the  errors  are  normally  distributed.  We  are  interested  here  in  whether  the 
error  distribution  is  skewed,  not  whether  it's  positively  or  negatively 
skewed.  Accordingly,  I  have  tabulated  the  absolute  value  of  skewness  for 
randomly-generated  errors,  allowing  us  to  use  it  in  a  two-tailed  test  of 
normality. 

B.5  Note  that  the  data  for  |GAMMA1|  and  GAMMA2  in  Tables  B.l  through  B.20 
extend  Geary  and  Pearson's  tables  [5]  for  the  skewness  and  kurtosis  of 
normally-distributed  data  to  smaller  sample  sizes.  The  sample  size  for  each 
table  is  equal  to  n(n-l)/2,  where  n  is  the  order  of  the  matrix. 
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Statistics  Addressing  Independence  of  Errors 


B.6  The  statistics  CHISQC  and  PC2C  address  the  independence  of  the  error 

2 

terms.  My  initial  approach  to  testing  independence  of  errors  used  a  X 

statistic  based  on  a  2x2  contingency  table.  This  approach  led  to  a  test  that 

was  little  different  from  one  based  on  the  signs  of  individual  error  terms  and 

2 

had  little  intuitive  appeal.  A  simple  X  test  that  compares  the  numbers  of 
positive  and  negative  error  terms  with  their  expected  values  was  substituted 
for  the  contingency  table  approach.  The  CHISQC  statistics  in  Tables  B.l 
through  B.20  are  based  on  the  latter  approach. 

2  o 

B.7  PC2C  is  the  X  probability  for  CHISQC.  If  CHISQC  is  X  -distributed  with 

one  degree  of  freedom,  PC2C  is  uniformly  distributed  between  zero  and  one,  and 

has  a  mean  of  0.5  and  a  variance  of  1/12.  Furthermore,  the  values  of  PC2C 

will  be  approximately  equal  to  the  percentages  in  the  left  column  of  each 

2 

table  if  CHISQC  is  X  -distributed.  The  tables  show  that  these  relationships 
do  not  hold  for  matrices  smaller  than  about  8x8.  This  occurs  because  the 
numbers  of  positive  and  negative  error  terms  are  integers,  so  only  discrete 
values  of  CHISQC  occur.  For  example,  it  is  easily  shown  that  for  3x3  SJMs, 
only  two  values  of  CHISQC,  1/3  and  3,  can  occur.  Their  probabilities  are 
0.75  and  0.25,  respectively,  under  the  assumption  that  positive  and  negative 
errors  are  equally  likely. 

B.8  Because  the  values  of  CHISQC  are  discrete,  especially  for  small  SJMs,  a 
X2  test  for  independence  based  on  the  numbers  of  positive  and  negative  error 
terms  will  yield  only  gross  indications  of  whether  the  error  terms  are  in¬ 
dependent.  A  better  test  would  use  the  values  of  CHISQC  given  in  the  tables. 

2 

However,  the  X  test  is  easier  to  implemenc,  so  it  was  selected  as  the  test  of 
independence  of  errors. 
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